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Overview 


Purpose  of  Research 

Model  based  systems  engineering  (MBSE)  is  becoming  increasingly  more  important  in  the 
practice  of  SE.  MBSE  methods  and  tools  are  used  throughout  the  entire  lifecycle  to  generate 
systems,  software  and  hardware  products,  and  work  towards  replacing  labor-intensive  and 
error-prone  documentation-based  processes  with  model-based  methods.  To  take  advantage  of 
model-based  techniques  to  develop  systems,  it  is  important  to  improve  human  and  technology 
integration  to  make  trades  and  decide  on  what  is  most  effective  given  the  present  knowledge 
and  future  uncertainties,  as  well  as  make  logical  decisions  based  on  the  availability  of  resources 
and  constraints.  The  Interactive  Model-Centric  Systems  Engineering  (IMCSE)  research  program 
will  develop  the  SE  methods,  processes  and  tools  to  improve  this  interaction,  with  the  goal  of 
accelerating  the  transition  of  SE  to  become  a  more  model-based  discipline. 

The  IMCSE  research  program  aims  to  develop  transformative  results  through  enabling  intense 
human-model  interaction,  to  rapidly  conceive  of  systems  and  interact  with  models  in  order  to 
make  rapid  trades  to  decide  on  what  is  most  effective  given  present  knowledge  and  future 
uncertainties,  as  well  as  what  is  practical  given  resources  and  constraints. 


Work  Accomplished  in  Phase  1 

The  IMCSE  research  program  presently  involves  three  projects  initiated  in  2014.  These  work 
accomplished  on  the  projects  in  this  phase  include: 

1.  Pathfinder  Project.  This  project  is  investigating  the  current  state  of  the  art/practice  in  IMCSE. 
The  research  team  has  been  conducting  informal  surveys  and  performing  literature  review  to 
establish  a  preliminary  understanding  of  what  is  being  done  in  practice  with  current  IMCSE- 
related  MPTs,  and  what  research  has/is  being  performed.  The  results  of  this  knowledge 
gathering  will  be  used  to  inform  an  invited  workshop,  focused  on  identifying  research 
opportunities,  gaps  and  issues  along  with  associated  priorities  and  initial  plans.  The  team  has 
performed  initial  planning  in  support  of  the  workshop,  and  has  looked  at  multiple  models  for 
documenting  the  results  of  the  workshop  in  a  report. 

2.  Interactive  Schedule  Reduction  Model.  The  research  team  initiated  its  plan  for  using  and 
improving  an  existing  prototype  system  dynamics  (SD)  model  to  interactively  explore 
alternatives  in  the  systems  development  process  and  application  of  resources.  The  model 
enables  rapid  sensitivity  analysis  of  various  factors  to  determine  their  potential  impact  on 
program  schedule,  and  investigates  new  methods  for  human  interaction  with  the  model.  The 
goal  of  this  year's  effort  is  to  develop  and  evaluate  exploratory  extensions  of  the  SD  model,  and 
to  prototype  an  interactive  interface  to  the  model  resulting  in  a  new  prototype  for  user  testing. 
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3.  Interactive  Epoch-Era  Analysis.  The  research  team  developed  a  strategy  for  extending  a 
current  approach  for  evaluating  systems  under  uncertainty,  Epoch-Era  Analysis  (EEA),  through 
the  development  of  an  interactive  capability.  The  goal  of  this  year's  effort  is  to  develop  a 
method  and  demonstration  prototype,  with  a  case  application.  This  case  application  will  serve 
as  a  pathfinder  for  identifying  key  considerations  for  applicability  and  deployability  of  the 
method  for  eventual  DoD  use.  It  will  also  inform  the  next  phase  effort  to  evolve  to  a  prototype 
for  user  testing. 

4.  Supporting  MPTs.  During  Phase  1,  the  opportunity  to  develop  several  MPTs  to  support 
IMCSE  arose,  including  supporting  infrastructure  (e.g.  databases),  software  (e.g.  IVTea  Suite), 
and  methods  (e.g.  Value  Model  Trading).  These  supporting  MPTs  are  essential  for  success  in 
the  three  research  projects,  as  well  as  to  help  with  capturing  and  transferring  knowledge  gained 
in  this  research  effort.  The  IVTea  Suite  software  and  the  Value  Model  Trading  demonstration 
case  are  described  in  this  report. 


Accomplishments  in  Phase  1 

The  following  findings  have  resulted  from  the  Phase  1  effort  over  the  4  month  period  of 
performance: 

1.  Knowledge  and  information  have  been  gathered  in  support  of  the  pathfinder  project, 
and  alternative  models  for  structuring  the  initial  workshop  and  a  resulting  report  were 
explored.  The  plan  for  the  Phase  2  workshop  was  developed. 

2.  Within  each  of  four  pillars  (key  topic  areas),  several  emerging  IMCSE-specific 
considerations  have  been  identified  through  literature  review  and  discussions  with 
subject  matter  experts.  At  the  intersection  of  the  pillars,  three  challenges  for  further 
investigation  have  been  identified:  tradeoff  of  models,  visual  analytics  of  artificial  data, 
and  perceptual  and  cognitive  considerations  in  human-model  interaction. 

3.  A  six  step  technical  approach  was  developed  for  evolving  an  existing  early  prototype  of  a 
interactive  schedule  reduction  model  to  include  a  user  interface  for  enhancing  user 
interaction  with  the  model.  Three  of  the  six  steps  have  been  implemented  in  Phase  1. 

4.  An  approach  for  an  interactive  Epoch-Era  Analysis  capability  was  formulated,  and 
supporting  techniques  and  tools  were  investigated.  Development  of  a  demonstration 
prototype  is  in  progress. 

5.  A  demonstration  case  for  value  model  tradeoffs  has  been  developed  for  a  Space  Tug 
system,  highlighting  methodological  considerations,  with  further  refinement  in  progress. 

6.  The  first  preliminary  version  of  Interactive  Value-driven  Tradespace  Exploration  and 
Analysis  (IVTea)  Suite  software  was  augmented  to  facilitate  IMCSE  research  and 
accelerate  the  application  of  techniques  and  case  studies. 


Research  Results 

The  research  team  has  produced  interim  research  outcomes  for  each  of  the  three  research 
thrusts  in  the  project:  foundations,  fundamentals,  and  applications.  These  outcomes  feed 
forward  into  Phase  2  of  the  project. 
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Next  Steps 


•  The  research  team  will  be  using  knowledge  and  information  gained  in  Phase  1  to  focus 
ongoing  efforts  in  Phase  2  to  further  explore  the  identified  IMCSE-related  considerations 
within  four  key  areas,  and  the  challenges  and  opportunities  at  their  intersection. 

•  The  pathfinder  workshop  plan  will  be  finalized  and  the  workshop  will  be  scheduled.  The 
workshop  will  be  held  and  a  workshop  report  will  be  published  and  released  to  elicit 
comments  and  recommendations.  Approaches  to  creating  a  broader  collaboratively-derived 
research  agenda  have  been  identified,  and  will  be  used  to  design  the  next  steps  in  building  a 
community  for  IMCSE  research. 

•  A  first  prototype  for  interactive  Epoch-Era  Analysis  will  be  completed  and  tested  with  a  case 
application,  along  with  preliminary  supporting  infrastructure,  which  will  then  be  used  to 
inform  the  design  of  a  next  version  prototype.  Specific  next  steps  are  described  on  page  47. 

•  The  team  will  continue  analysis  of  the  value  model  trades  in  the  demonstration  case,  along 
with  developing  a  more  complete  framework  and  process  for  how  to  conduct  value  model 
trades  more  generally.  Specific  next  steps  are  described  on  page  60. 

•  The  IVTea  Suite  will  continue  to  undergo  refinement  of  user  interface,  data  handling,  as  well 
as  development  of  additional  widgets  that  support  ongoing  research.  Specific  next  steps 
are  described  on  page  69. 

•  The  extended  interactive  schedule  reduction  model  prototype  will  be  completed  and  made 
available  for  user  testing.  Specific  next  steps  are  described  on  page  80. 

•  The  research  team  will  use  the  results  of  Phase  1  to  develop  several  publishable  papers  for 
the  CSER2015. 
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Year 


Focus 


Key  Deliverables 


Pre-2014  New  start 

Pathfinder  project  with  collaborative 
research  discovery;  exploratory 
extensions  to  an  existing  development 
schedule  reduction  model;  exploratory 
piloting  and  interactive  extensions  to 
Epoch-Era  method 

Pathfinder  Project:  Investigation  of  current  state  of 
art/practice.  Workshop  to  explore  issues  and 
opportunities,  with  report  on  workshop  results. 
Pathfinder  project  report,  with  findings  of  research 
opportunities,  gaps,  and  issues.  Out-year  research 
plans  based  on  pathfinder  results. 

Interactive  Schedule  Reduction  Model:  Exploratory 
extensions  implemented  and  evaluated.  Report  on 
exploratory  schedule  model.  Prototype  model  for  pilot 
application. 

Interactive  Epoch-Era  Analysis:  Exploratory  research  to 
develop  Interactive  capability,  with  demonstration  via  a 
mission  planning  support  application  case.  Report  on 
exploratory  research  and  case  application. 

Initiate  multi-year  research  plans  based 
on  pathfinder  results,  including  2014 

2015  project  follow-on  for  one  or  both  of  the 

exploratory  research  projects.  Assess 
results  individually  and  comparatively. 

IMCSE  Project  Applications:  Based  on  pathfinder  project 
results,  select  and  initiate  one  or  more  additional 
projects,  and  increase  SERC  member  collaboration  in 
projects.  Report  to  document  the  maturation  of  the 
MPTs  for  each  of  these  projects,  with  comparative 
results. 

Increasing  maturation  of  IMCSE  MPTs 
and  enabling  environments,  leading  to 
adoption  by  user  community  and 
assessment  of  real-world  impact; 
extend  IMCSE  scope  via  increased 

2016 

collaboration  of  additional  universities 
and  broader  user  community. 

Exploration  of  further  new-idea 
projects. 

IMCSE  MPT  Implementations  and  Impact  Assessments: 
Continued  maturation  and  implementation  of  IMCSE 
MPTs,  with  enabling  environments.  Ongoing  study  of 
impacts  resulting  in  a  comprehensive  report  of 
progress,  results,  and  opportunities. 

Increasing  maturation  and  synthesis  of 

IMCSE  MPTs  and  enabling 

environments,  leading  to  adoption  by 

user  community  and  demonstration  of 
2017-2018  ,  ,, 

real-world  impact;  sustain  and  increase 

collaboration  of  additional  universities 

and  broader  user  community.  Step-ups 

of  new-idea  projects. 

IMCSE  MPT  Synthesis  Impact  and  Effective  Practice 
Assessments:  Continued  maturation,  synthesis  and 
implementation  of  IMCSE  MPTs,  with  enabling 
environments.  Ongoing  study  of  real-world  impacts  to 
identify  successful  practices.  A  comprehensive  report  of 
impacts  and  insights,  with  guidance  on  practice. 

Figure  1.  IMCSE  Project  Timeline 
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Introduction 


The  IMCSE  research  program  aims  to  develop  transformative  results  through  enabling  intense 
human-model  interaction,  to  rapidly  conceive  of  systems  and  interact  with  models  in  order  to 
make  rapid  trades  to  decide  on  what  is  most  effective  given  present  knowledge  and  future 
uncertainties,  as  well  as  what  is  practical  given  resources  and  constraints. 


Motivation 

Models  have  significantly  changed  systems  engineering  practice  over  the  past  decade.  Most 
notably,  model-based  systems  engineering  (MBSE)  methods  and  tools  are  increasingly  used 
throughout  the  entire  system  lifecycle  to  generate  systems,  software  and  hardware  products, 
replacing  labor-intensive  and  error-prone  documentation-based  processes  with  model-based 
ones.  While  substantial  benefits  have  been  achieved,  the  most  impactful  application  of  models 
in  systems  engineering  has  yet  to  be  realized.  Models  are  needed  to  inform  engineering 
decisions.  Truly  transformative  results  will  only  come  through  intense  human-model 
interaction,  to  rapidly  conceive  of  systems  and  interact  with  models  in  order  to  make  rapid 
trades  to  decide  on  what  is  most  effective  given  present  knowledge  and  future  uncertainties,  as 
well  as  what  is  practical  given  resources  and  constraints. 

As  cited  in  the  SERC  2014-2018  Technical  Plan,  reports  have  found  significant  insufficiencies  in 
the  current  practice. 

The  National  Research  Council’s  “Human-System  Integration  in  the  System 
Development  Process,  ”  (NRC,  2007),  “Pre-Milestone  A  and  Early-Phase  SE,  ” 

(NRC,  2008),  and  “Critical  Code,  ”  (NRC,  2010)  studies  consistently  found  that 
the  SE  MPTs  for  integrating  hardware  engineering,  human  factors  engineering, 
and  software  engineering  into  a  scalable,  unified  approach  were  not  up  to  the 
challenges  of  the  complexity,  scale,  and  dynamism  characterizing  DoD  ’s  large- 
scale  systems  and  systems  of  systems. 

This  research  project  addresses  the  SERC's  Systems  Engineering  and  Systems  Management 
Transformation  (SEMT)  grand  challenge: 

Move  the  DoD  community ’s  current  systems  engineering  and  management  MPTs 
and  practices  away  from  sequential,  single  stovepipe  system,  hardware-first, 
outside-in,  document-driven,  point-solution,  acquisition-oriented  approaches; 
toward  concurrent,  portfolio  and  enterprise -oriented,  hardware -software -human 
engineered,  balanced  outside-in  and  inside-out,  model-driven,  set-based,  full  life 
cycle  approaches.  These  will  enable  much  more  rapid,  concurrent,  flexible, 
scalable  definition  and  analysis  of  the  increasingly  complex,  dynamic,  multi¬ 
stakeholder,  cyber-physical-human  DoD  systems,  systems  of  systems,  portfolios  of 
systems,  and  enterprises  of  the  future. 
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Insufficiencies  in  Current  Practice 

Early  concept  decisions  have  always  been  critically  important,  and  with  continuously  evolving 
systems  of  systems  having  long  life  spans,  such  decisions  are  now  made  throughout  the  entire 
life  cycle.  Soft  factors  become  increasingly  influential.  For  example,  trust  in  model-based  data 
sets  and  decisions  are  in  part  determined  by  the  chosen  model  itself  as  perceived  by  specific 
decision  makers.  The  timescale  of  making  early  architectural  decisions  is  out  of  sync  with  the 
current  model-based  systems  engineering  capabilities  and  decision  environments.  New 
algorithms  and  novel  modeling  approaches  must  be  discovered  to  accelerate  technical  and 
programmatic  decision  support  from  months  to  minutes.  In  order  to  effectively  leverage  and 
incorporate  human  knowledge  and  judgment,  an  interactive  capability  is  needed.  Much 
potential  exists  in  maturing  emerging  novel  methods  for  evaluating  system  responsiveness 
under  complex  uncertainties,  to  enable  engineering  of  resilient  systems. 

Relevant  Prior  SERC  Research 

IMCSE  will  include  and  significantly  extend  the  traditional  focus  on  the  modeling  of  system 
products  and  the  use  of  the  models.  Extensions  will  address  the  modeling  of  system  execution 
processes,  such  as  operational  concept  formulation,  and  system  development  processes,  which 
can  also  be  executed  to  aid  in  the  generation  of  system  products.  As  emphasized  in  the  SERC 
Systems  2020  Report,  an  additional  focus  on  modeling  the  system's  environment  will  be 
pursued,  which  is  needed  for  performing  many  of  the  ilities  tradespace  and  affordability 
analyses.  Models  can  also  improve  affordability  by  automatically  generating  needed 
documentation,  or  even  better  by  serving  as  the  documentation  itself.  Further,  models  can 
reduce  or  avoid  system  overruns  and  performance  shortfalls  by  enabling  more  thorough 
Analyses  of  Alternatives  and  evidence-based  decision  reviews.  Modeling  the  system's  dynamic 
operational  environment  remains  an  open  area  of  research.  IMCSE  has  a  relationship  to  many 
of  the  past  and  ongoing  SERC  projects.  Several  of  the  most  relevant  prior  SERC  projects  are 
summarized  in  Appendix  A. 
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IMCSE 


Interactive  Model-centric  Systems  Engineering  (IMCSE),  not  to  be  confused  with  Model-based 
Systems  Engineering  (MBSE),  is  a  research  program  that  seeks  to  encourage  the  development 
of  augmented  complex  systems  thinking  and  analysis  to  support  data-driven  decision  making. 


What  is  IMCSE? 

Systems  scientists  have  long  recognized  that  humans  possess  unique  abilities  for  anticipation 
rather  than  simple  reactive  response.  In  order  to  increase  the  likelihood  of  developing  complex 
systems  that  can  deliver  value  to  stakeholders  across  a  dynamic,  uncertain  future,  systems 
engineers  must  have  both  reactionary  and  anticipatory  capacity  to  make  better  decisions.  In 
contrast  to  reactionary  capacity,  which  involves  developing  solutions  after  the  fact,  anticipatory 
capacity,  as  defined  by  Rhodes  and  Ross  (2009),  is  "the  capacity  to  continuously  develop  and 
apply  knowledge  acquired  through  a  structured  approach  to  anticipate  1)  changing  scenarios  as 
stakeholder  needs  and  systems  context  change  over  time;  2)  to  consider  their  consequences; 
and  3)  to  formulate  design  decisions  in  response^.  Three  key  enablers  of  anticipatory  capacity 
are  mindset,  methods,  and  environment.  Models  represent  an  abstraction  of  reality  in  order  to 
make  predictions  about  the  future.  Models  can  come  in  a  variety  of  forms  and  formats,  but 
fundamentally  they  are  an  encapsulation  of  reality  that  humans  use  to  augment  their  ability  to 
make  sense  of  the  world  and  anticipate  future  outcomes.  Improvements  in  computation, 
simulation  technologies,  and  human-machine  interaction  have  created  an  opportunity  to 
enable  human-model  interaction  to  greatly  enhance  anticipatory  capacity.  Complex,  integrated 
models,  of  various  levels  of  fidelity,  can  create  large  data  sets  in  need  of  human  pattern 
recognition  skills.  Interaction  enables  real  time  interrogation  of  the  data  and  opportunities  for 
model  creation  as  well  as  validation  and  learning.  IMCSE  is  a  research  program  intended  to 
leverage  human-model  interaction  in  order  to  transform  systems  engineering  decision  making 
through  anticipatory  capacity. 


Research  Program  Vision 

The  vision  for  the  IMCSE  research  program  is  to  develop  transformative  results  through 
enabling  intense  human-model  interaction,  to  rapidly  conceive  of  systems  and  interact  with 
models  in  order  to  make  rapid  trades  to  decide  on  what  is  most  effective  given  present 
knowledge  and  future  uncertainties,  as  well  as  what  is  practical  given  resources  and 
constraints. 

In  order  to  accomplish  this  vision,  IMCSE  will  pursue  a  balanced  basic  and  applied  research 
approach.  This  will  leverage  the  strength  of  the  academic  environment  (e.g.  developing 
fundamentals,  approaching  with  rigor,  providing  a  neutral  third  party  view  of  the  problem). 
Additionally,  IMCSE  will  strive  to  keep  the  research  relevant  to  the  sponsor  community,  as  well 
as  enabling  opportunities  for  knowledge  and  methods,  processes,  and  tools  (MPTs)  transfer  to 

^  Rhodes,  D.H.  and  Ross,  A.M.,  "Anticipatory  Capacity:  Leveraging  Model-Based  Approaches  to  Design  Systems  for 
Dynamic  Futures,"  2nd  Annual  Conference  on  Model-based  Systems,  Haifa,  Israel,  March  2009. 
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sponsors.  Such  knowledge  transfer  opportunities  include  workshops,  teleconferences  and 
meetings,  reports,  papers,  collaboration  with  other  SERC  activities,  prototypes,  methods, 
processes,  and  tools  (MPTS),  government  partner  applications,  and  potential  student 
internships. 


IMCSE  Pillars -Four  Topic  Areas 

IMCSE  is  motivated  by  the  convergence  of  four  key  topic  areas:  big  data,  visual  analytics, 
complex  systems,  and  model-based  systems  engineering.  Each  of  these  areas  have  associated 
with  them  large  research  and  application  efforts.  This  research  program  seeks  to  identify 
synergies  and  gaps  at  the  intersection  of  these  four  topic  areas,  and  leverage  existing  and  new 
techniques  in  this  area  to  create  new  knowledge  and  capabilities  for  systems  engineering 
decision  making.  In  order  to  focus  the  research  program,  early  efforts  are  aimed  to  identify  key 
challenges  that  summaries  the  gaps  in  the  existing  topic  area  overlaps. 


Big  Data 

We  live  in  a  world  with  big  data.  As  data  storage  costs  have  shrunk,  so  too  has  the  need  for 
purging  data.  Additionally  data  is  being  generated  through  a  large  and  growing  number  of 
means,  from  sensors  to  users  to  corporate  IT  environments.  Even  "document-based"  data  is 
becoming  digital  as  technology  (including  OCR)  becomes  commonplace  for  capturing  physical 
information  as  digital  data.  No  consensus  currently  exists  regarding  a  formal  definition  on  what 
constitutes  "big  data,"  but  it  is  generally  recognized  as  having  a  number  of  characteristics  that 
make  it  "big."  One  example  description,  from  IBM^,  characterizes  big  data  as  having  challenges 
regarding  Volume,  Variety,  Velocity,  and  Veracity.  The  challenge  for  Volume  revolves  around 
the  scale  of  the  data  (e.g.  how  to  store  and  recall  large  numbers  of  field  entries  in  a  data  base?). 
The  challenge  for  Variety  revolves  around  the  different  forms  of  data  (e.g.  how  to  store  and 
compare  data  from  photos,  videos,  blogs,  articles,  etc.?).  The  challenge  for  Velocity  revolves 
around  the  analysis  of  streaming  data  (e.g.  how  to  account  for  and  parse  large  streams  of 
potentially  incomplete  data  in  real  time?).  The  challenge  for  Veracity  revolves  around  the 
uncertainty  of  the  data  (e.g.  different  data  sources  have  different  degrees  of  trustfulness  and 
reliability,  so  how  to  fuse  data  from  such  sources?). 

The  impact  of  big  data  is  being  felt  across  many  fields  from  transportation  to  entertainment, 
education  to  banking,  which  will  only  increase  as  the  benefit  of  leveraging  such  data  becomes 
apparent.  Such  benefits  have  been  recognized  by  a  growing  number  of  commercial 
organizations  who  are  leveraging  this  inundation  of  data  to  gain  insights  into  phenomena  to 
create  predictive  models  (e.g.  of  user  behavior  and  preferences).  For  example,  Amazon  and 
Netflix  both  have  sophisticated  user  preference  models  that  are  used  to  make 
recommendations  to  users  based  on  their  own  (and  related  others)  browsing  and 
shopping/viewing  history.  Additionally,  Netflix  has  used  this  information  (and  Amazon  recently 
as  well)  to  generate  design  requirements  for  new  shows.  House  of  Cards,  produced  by  Netflix, 


^  http://www.ibmbigdatahub.com/sites/default/files/infographic  file/4-Vs-of-big-data.ipg 
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was  partially  designed  based  on  derived  preferences  of  its  viewer  base  in  order  to  increase  the 
perceived  value  of  the  program^. 

While  not  necessarily  generated  in  a  similar  manner,  DoD  has  already  a  vast  amount  of  data 
stored  in  documents,  for  example  requirements  documents,  design  documents,  DoDAF,  etc, 
which  represent  latent  data  that  could  be  leveraged  using  techniques  being  developed  in  the 
commercial  application  space.  What  would  a  ground  vehicle  recommendation  look  like?  How 
would  it  parse  and  analyze  historical  requirements  documents  and  contextual  information  in 
order  to  predict  and/or  augment  modern  user  needs?  Big  data  is  a  topic  area  that  holds 
promise  in  providing  a  foundation  for  large  scale  analytics  to  predict  the  future. 

Visual  Analytics 

Visual  analytics  is  a  topic  area  that  has  likewise  been  a  growing  area  for  research  and 
application.  At  its  core,  visual  analytics  is  about  collaboration  between  human  and  computer 
using  visualization,  data  analytics,  and  human-in-the-loop  interaction.  More  than  just 
visualization  tools,  visual  analytics  aims  to  take  advantage  of  a  human's  ability  to  discover 
patterns  and  drive  inquiry  in  order  to  make  sense  of  data.  In  2007,  DHS  sponsored  the  National 
Visualization  and  Analytics  Center,  which  developed  a  research  agenda  called  Illuminating  the 
Path.  In  it,  visual  analytics  was  defined  as  "the  science  of  analytical  reasoning  facilitated  by 
interactive  visual  interfaces"  that  "provides  the  last  12  inches  between  the  masses  of 
information  and  the  human  mind  to  make  decisions."^  Application  areas  range  from  homeland 
security  to  anti-fraud,  banking  to  insurance.  One  common  element  in  much  of  the  current 
visual  analytics  work  involves  case  applications  comparing  VA-supported  inquiry  results  to 
ground  truth,  that  is,  discovery  of  patterns  in  "natural"  data.  One  consequence  of  these  studies 
is  that  the  validity  of  the  applications  can  be  compared  to  observable  "truth."  This  allows 
researchers  to  test  how  well  their  predictive  models  match  reality,  for  example,  using  VA  to 
discover  hackers  trying  to  break  into  streams  of  ATM  data;  or  discovering  patterns  of  use  in  bike 
sharing  programs  as  a  function  of  time  and  geography.  In  both  of  these  examples  there  are 
"real"  processes  at  play  and  actual  measurable  real  world  data  against  which  to  validate 
predictions  by  the  human-machine  VA  system.  VA  has  been  shown  to  be  incredibly  useful  for 
developing  models  of  natural  data. 

Complex  Systems 

Our  application  domain  is  the  development  of  (artificial)  systems  that  serve  the  purpose  of 
delivering  value  to  stakeholders.  By  "artificial"  we  mean  that  these  systems  are  artifacts 
created  by  humans  for  a  purpose,  to  be  contrasted  with  natural  systems,  which  are  not  created 
by  humans.  Over  time,  the  complexity  of  systems  has  tended  to  grow,  not  only  due  to  scale 
and  interconnectedness,  but  also  due  to  increased  scope  in  our  ability  to  describe  the  system. 
This  enhanced  scope  reflects  realization  that  the  success  of  artificial  systems  requires  a  fuller 


^http://www.nvtimes.com/2013/02/25/business/media/for-house-of-cards-using-big-data-to-guarantee-its- 

popularitv.html?pagewanted=all&  r=0 

Jim  Thomas,  Director,  USDHS  National  Visualization  and  Analytics  Center,  "Visual  Analytics:  An  Agenda  in 
Response  to  DHS  Mission  Needs,"  2007 
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understanding  of  how  the  system  is  structured,  behaves,  performs  in  different  contexts, 
performs  over  time,  is  perceived  across  stakeholders^  This  means  that  to  describe  a  complex 
system,  one  must  consider  all  five  perspectives,  thereby  creating  a  richer  description  of  the 
system.  Developing  complex  systems  necessitates  an  approach  to  generate,  manage,  and 
analyze  artificial  data  across  these  five  aspects. 

Model-Based  Systems  Engineering  (MBSE) 

Traditional  systems  engineering  has  been  document-heavy  and  process-driven,  resulting  in 
many  opportunities  for  miscommunication  and  mistakes  during  "hand-offs"  between  phases 
and  teams.  Models  are  often  used  during  design  and  development  in  order  to  predict  behavior 
or  other  consequences  of  design  decisions,  before  the  system  is  built  or  operated.  In  contrast  to 
document-based  engineering,  "model-based  systems  engineering  (MBSE)  is  the  formalized 
application  of  modeling  to  support  system  requirements,  design,  analysis,  verification,  and 
validation  activities  beginning  in  the  conceptual  design  phase  and  continuing  throughout 
development  and  later  life  cycle  phases."^  Today,  however,  standalone  models  are  typically 
related  through  documents.  A  future  vision  is  for  organizations  to  use  "shared  system  model(s) 
with  multiple  views,  and  connected  to  discipline  models,"  in  order  to  reduce  effort  creating  and 
aligning  documents,  and  to  increase  synthesis  and  coherence  across  disciplines  throughout 
design^.  Regardless  of  the  degree  to  which  MBSE  is  employed,  its  benefits  stem  from  moving  to 
models  to  represent  systems  with  less  ambiguity,  more  parsimony,  and  more  consistency, 
resulting  in  reduced  acquisition  time,  enhanced  reliability,  etc.  MBSE  generates  "artificial  data" 
about  systems  which  can  be  used  to  make  decisions  that  impact  the  future  and  continuing 
success  of  that  system. 

Synthesizing  the  Pillars 

Each  of  the  four  topic  areas  above  are  themselves  large  areas  of  active  research  and 
development  across  government,  academia,  and  industry.  IMCSE  in  particular  is  interested  in 
the  intersection  of  these  four  areas  with  application  to  improving  systems  engineering  decision 
making.  More  than  just  applied  visual  analytics,  IMCSE  seeks  to  look  at  data  generated  by 
models,  in  order  to  make  better  decisions  in  how  to  deliver  sustained  value  to  stakeholders.  In 
particular  preliminary  investigation  has  uncovered  two  initial  challenges  for  IMCSE  to  address. 

These  include: 

1)  Visual  analytics  of  artificial  (i.e.  model-generated)  data:  how  does  this  differ  from  VA  of 
natural  data?  How  to  take  into  account  the  impact  of  various  model  implementations  on 


^  Rhodes,  D.H.,  and  Ross,  A.M.,  "Five  Aspects  of  Engineering  Complex  Systems:  Emerging  Constructs  and  Methods," 
4th  Annual  IEEE  Systems  Conference,  San  Diego,  CA,  April  2010 

®  Caspar,  H.,  Rhodes,  D.H.,  Ross,  A.M.,  and  Erikstad,  E.O.,  "Addressing  Complexity  Aspects  in  Conceptual  Ship 
Design:  A  Systems  Engineering  Approach"  Journo/  of  Ship  Production  and  Design,  Vol.  28,  No.  4,  Nov  2012,  pp.  145- 
159. 

^  INCOSE  SE  Vision  2020  (INCOSE-TP-2004-004-02,  Sep  2007) 

^  http://www.omgwiki.org/MBSE/doku.php?id=mbse:incose_mbse_iw_2014 
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pattern  finding  and  matching  of  mental  and  constructed  models?  How  to  validate 
predictions  without  ground  truth  available? 

2)  Active  tradeoffs  of  models  themselves:  too  often  models  are  used  without  sufficient 
investigation  into  the  impact  of  the  models  on  the  data  being  used  for  decisions;  these 
include  performance  models,  cost  models,  and  value  models.  Model  selection 
fundamentally  impacts  the  patterns  to  be  discovered  in  the  artificial  data. 

Ultimately,  the  goal  of  IMCSE  is  to  leverage  visual  analytics  applied  to  model-generated  "big 
data,"  in  order  to  develop  a  rigorous  framework,  with  associated  methods,  processes,  and  tools 
(MPTS),  which  will  result  in  transformative  new  capabilities  for  complex  systems  engineering 
decision  making. 


IMCSE  Approach 

IMCSE  uses  three  complimentary  thrusts  with  different  timescales,  in  order  to  have  impact  on 
the  long  term,  the  near  term,  and  the  present. 

These  thrusts  include: 

•  Foundations:  1  year,  set  the  stage  for  IMCSE  for  long  term  impact 

•  Fundamentals:  multi-year,  medium  timescale  impact,  potentially  broad  applicability 

•  Applications:  1  year,  short  timescale  impact,  generate  deployment  opportunities 

Current  progress  in  each  of  these  three  thrusts  will  be  described  in  the  following  sections  of  the 
report. 
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Foundations 


The  foundations  research  thrust  is  currently  focused  on  two  activities.  The  first  is  the  research 
pathfinder  project,  including  the  initial  'setting  the  stage'  activity  of  an  invited  workshop. 
Extending  the  results  of  the  workshop,  a  more  extensive  effort  will  build  a  community  of 
interest.  The  end  result  will  be  a  collaboratively-derived  research  agenda.  The  second  activity 
is  investigating  the  current  state  practice  and  emerging  state  of  the  art.  This  includes  literature 
review  and  discussions  with  subject  matter  experts.  Ongoing  results  will  inform  the  research 
agenda,  and  the  specific  projects  undertaken  in  the  fundamentals  and  applications  thrusts. 


IMCSE  Research  Pathfinder  Project 

A  pathfinder  project  brings  together  the  relevant  stakeholders  to  develop  a  research  vision  and 
research  priorities,  and  a  roadmap  to  achieve  them.  The  IMCSE  pathfinder  project  will  include 
one  or  more  face-to-face  gatherings  of  stakeholders  in  the  research  agenda,  as  well  as 
specifically  focused  research  meetings.  Given  the  footprint  of  IMCSE,  it  would  not  be  possible 
to  convene  a  large  enough  community  in  a  participant  workshop  for  the  purpose  of  a 
collaboratively-derived  research  agenda.  Our  research  team  is  looking  at  various  approaches 
that  have  been  used,  and  defining  pathfinder  efforts  to  leverage  the  success  of  these 
approaches.  The  goal  is  to  be  able  to  engage  a  large  and  diverse  community  around  the 
research  agenda,  and  determine  an  approach  that  may  include  both  face-to-face  and  virtual 
activities. 


Launching  THE  Pathfinder  Project 

Preliminary  efforts  in  defining  a  research  vision  and  results  of  exploratory  knowledge  gathering 
are  being  used  to  develop  the  plan  for  conducting  an  initial  pathfinder  research  workshop.  The 
research  team  is  in  the  process  of  selecting  invited  participants  for  a  small  initial  workshop,  to 
be  held  in  Phase  2  of  this  project.  The  team  has  performed  initial  planning  in  support  of  the 
workshop,  and  has  looked  at  multiple  models  for  documenting  the  results  of  the  workshop  in  a 
report.  The  outcome  of  the  workshop  event  will  be  a  workshop  report.  The  goal  is  to  identify 
high  level  research  needs  and  questions,  along  with  identifying  gaps,  issues  and  opportunities. 
The  results  of  the  workshop  will  be  made  available  for  review  and  comment,  and  the  research 
report  will  be  updated  with  the  added  information.  The  goal  of  the  workshop  and  resulting 
report  is  to  seed  the  larger  effort  to  build  a  community  of  interest  and  undertake  a  more 
extensive  research  pathfinder  activity. 

The  pathfinder  activities  in  Phase  1  and  Phase  2  of  this  research  project  will  inform  efforts  in 
Phase  3  to  elicit  information  on  state  of  the  art  and  practice,  identify  additional  research 
stakeholders,  clarify  and  expand  the  urgent  research  questions,  and  investigate  priorities.  The 
Phase  3  objective  will  be  to  establish  a  collaboratively-derived  IMCSE  research  agenda.  The 
ultimate  goal  is  to  build  a  community  of  interest  around  the  IMCSE  research  agenda,  build 
partnerships  for  research,  and  to  foster  collaboration  in  addressing  the  emerging  challenges  at 
the  intersection  of  the  four  topic  areas. 
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Building  a  Community  of  Interest 


Each  of  the  four  topic  areas  (big  data,  visual  analytics,  complex  systems,  and  model-based 
systems  engineering)  engages  researchers  from  multiple  disciplines  and  domains.  IMCSE 
research  seeks  to  encourage  the  development  of  augmented  complex  systems  thinking  and 
analysis  to  support  data-driven  decision  making.  The  stakeholders  who  contribute  to  and 
benefit  from  IMCSE  include  government  sponsors,  senior  decision  makers,  system  designers, 
analysts,  academic  researchers,  policy  makers,  funding  agencies  and  others.  Bringing  such  a 
community  together  around  a  shared  research  vision  and  agenda  is  a  significant  challenge,  but 
there  are  prior  exemplars. 

During  this  phase,  the  research  team  has  been  investigating  successful  efforts  in  other  fields  to 
create  a  research  agenda  through  a  collaborative  approach  involving  a  large  and  diverse  set  of 
participants.  A  particular  feature  of  these  exemplars  is  the  success  in  bringing  together 
stakeholder  from  government,  non-governmental  organizations,  academia  and  industry  to 
narrow  the  gap  between  data  generated  in  research  and  the  information  required  by  policy 
makers.  One  recent  effort  was  the  development  of  a  collaboratively-derived  science-policy 
research  agenda,  driven  by  the  need  for  policy  makers  to  understand  science  and  for  scientists 
to  understand  policy  processes.®  Participants  were  selected  to  cover  a  wide  range  of  disciplines 
and  constituencies.  Each  participant  submitted  a  list  of  questions,  resulting  in  239  questions  in 
the  first  stage.  A  process  of  voting,  deliberation  and  further  voting  resulted  in  a  final  set  of  40 
questions,  and  then  grouped  thematically  into  six  groups.  The  authors,  Sutherland  et  al.  (2012), 
noted  the  outcome  is  inevitably  influenced  by  the  composition  of  the  participants  and  the 
process.  While  not  'reproducible,  it  is  highly  likely  this  approach  would  yield  similar  emergent 
general  themes. 

Ingram,  et  al.  (2013)^°  applied  the  collaboratively-derived  research  approach  of  Sutherland  et 
al.  (2012)  in  addressing  questions  related  to  the  UK  food  security  and  food  system.  They  found 
it  proved  useful  for  engaging  a  wide  range  of  stakeholders  and  helped  establish  a  well-balanced 
discussion  on  the  production  system  and  the  security  outcomes,  informing  a  research  agenda 
from  public  funders  and  applied  industry  viewpoints,  as  well  as  mapping  needs  onto  the 
international  food  security  agenda.  The  dimensions  in  this  work  are  not  unlike  IMCSE,  where 
there  are  intertwined  needs  of  defense  funding  agencies,  system  developers,  and  impacts  to 
national/international  security. 

Sutherland,  et  al.  (2011)^^  discuss  methods  that  "maximize  inclusiveness  and  rigour  in  such 
exercises  include  solicitation  of  questions  and  priorities  from  an  extensive  community,  online 
collation  of  material,  repeated  voting  and  engagement  with  policy  networks  to  foster  uptake 


^  Sutherland,  W.,  et  al.,  "A  Collaboratively-Derived  Science-Policy  research  agenda",  PLoS  ONE,  Vol.  7,  Issue  3, 
March  2012 

Ingram,  J.,  "Priority  research  questions  for  the  UK  food  system".  Food  Sec.,  (2013)  5:617-636. 

Sutherland,  W.  J.,  Fleishman,  E.,  Mascia,  M.  B.,  Pretty,  J.  and  Rudd,  M.  A.  (2011),  Methods  for  collaboratively 
identifying  research  priorities  and  emerging  issues  in  science  and  policy.  Methods  in  Ecology  and  Evolution,  2:  238- 
247 
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and  application  of  the  results".  These  authors  summarize  eight  exercises  with  variation  on  the 
general  approach.  Their  work  in  bridging  the  gap  between  scientific  researchers  and  policy 
makers  is  notable,  as  IMCSE  needs  to  bridge  the  gap  between  the  engineer/scientists  and  the 
senior  decision  makers  and  policy  makers.  While  the  authors  work  in  a  different  domain  of 
interest,  their  work  has  resulted  in  a  set  of  guiding  principles  for  generating  a  collaboratively- 
derived  research  agenda.  The  guidance  covers  defining  the  project;  organizing  the  participants; 
soliciting  and  managing  questions  or  issues;  voting  systems;  and  disseminating  results. 
Experiences  and  effective  practices  in  establishing  collaboratively-derived  research  agendas 
provide  insight  and  guidance,  with  potential  to  enhance  the  success  of  the  pathfinder  project 
for  IMCSE. 

Early  in  Phase  2,  the  research  team  will  develop  its  specific  approach  for  generating  a 
collaboratively-derived  research  agenda,  to  build  on  the  outcomes  of  the  initial  pathfinder 
workshop. 
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Exploring  the  IMCSE-relevant  State  of  the  Art  and  Practice 


In  Phase  1,  the  research  team  initiated  its  literature  review  and  knowledge  gathering  to  explore 
the  state  of  the  art  and  practice,  specifically  as  related  to  the  IMCSE  area.  The  team's 
organizing  framework  for  investigation  is  around  four  key  topic  areas,  as  well  as  the  emerging 
challenges  at  their  intersections.  The  four  topic  areas  are  (1)  big  data,  (2)  visual  analytics,  (3) 
complex  systems  and  (4)  model-based  systems  engineering.  These  four  areas  have  an 
extensive  and  expanding  landscape,  and  the  goal  of  our  research  is  not  to  establish  a 
comprehensive  state  of  the  art  and  practice  of  the  topic  areas,  but  rather  to  discover  the  critical 
themes,  challenges  and  questions  that  are  directly  relevant  for  IMCSE. 

During  Phase  1,  three  challenges  at  the  intersections  emerged:  (1)  tradeoff  of  models;  (2)  visual 
analytics  of  artificial  (model-generated)  data;  and  (3)  perceptual  and  cognitive  considerations  in 
human-model  interaction,  feed  these  into  the  pathfinder  workshop  activity.  During  the 
workshop,  it  is  expected  that  additional  cross-cutting  considerations  will  be  identified. 

In  the  following  subsections,  we  highlight  several  of  the  themes  within  each  of  the  four  topic 
areas  and  the  three  intersection  topics. 


Big  Data 

Big  data  provides  a  foundation  for  large  scale  analytics  to  predict  the  future. 

Big  data  provides  a  foundation  for  large-scale  analytics  to  predict  the  future  across  domains  as 
diverse  as  defense,  healthcare,  and  urban  planning.  Yet  as  evolving  technological  capabilities 
allow  for  the  capture,  management,  and  exploration  of  increasingly  large  and  complex  data 
sets,  researchers  are  also  faced  with  new  and  emerging  methodological  questions  when 
grappling  with  the  forecasting  implications  of  big  data.  Broadly  speaking,  two  of  the  most 
significant  issues  facing  researchers  in  the  field  of  big  data  today  are  trust  in  the  data  and 
representativeness  of  the  models  they  engender. 

Overprojection  of  trend  models.  Perhaps  one  of  the  clearest  examples  highlighting  both 
the  promises  and  pitfalls  of  big  data  driven  analytics  is  the  recent  Google  Flu  Trends 
(GFT)  project,  which  aimed  to  "nowcast"  flu  prevalence  based  on  the  real-time 
tabulation  of  query  entries.  In  the  domain  of  global  public  health,  big  data  offers  the 
possibility  not  only  of  facilitating  the  epidemiological  tracking  of  disease,  but  of 
forecasting  the  spread  of  disease  as  well,  thereby  allowing  for  the  effective  and  timely 
distribution  of  critical  resources  such  as  medication  and  aid  workers.  Yet  projections 
offered  by  the  GFT  wildly  overestimated  incidence  of  influenza  in  the  US,  when 
compared  against  doctors'  reports  collected  by  the  Center  for  Disease  Control  and 
Prevention.  Recent  analyses  of  this  failure  in  big  data  projection  have  revealed  systemic 
problems  in  the  use  of  such  data  sets  in  forecasting  models. Specifically, 


Lazer,  D.,  Kenney,  R.,  King,  G.,  &  Vespignani,  A.  (2014).  The  parable  of  Google  Flu:  traps  in  Big  Data  analysis. 
Science,  343(6176):  1203-1205. 
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overprojection  of  trend  models  was  seen  to  result  from  failures  to  consider  the 
uniqueness  of  individual  data  point.  In  the  case  of  the  GFT,  a  cluster  of  regionalized 
queries  probing  flu  symptomology  could  underlie  a  local  outbreak  as  much  as  it  could 
coincide  with  the  theme  of  a  district  school's  science  fair. 

Misinterpretation  of  a  correlational  relationship  to  mean  a  causal  connection.  In  this 
regard,  the  interpretation  of  big  data  is  vulnerable  to  one  of  the  hallmark  errors  of 
shoddy  statistics— the  misinterpretation  of  a  correlational  relationship  to  mean  a  causal 
connection.  In  short,  when  researchers  fail  to  consider  the  epistemological 
heterogeneity  of  the  individual  data  points  within  a  very  large  set— that  is,  the  meaning 
underlying  the  behavioral  actions  which  have  been  tabulated  and  accrued— their  failure 
to  engage  with  the  ambiguity  inherent  in  the  data  set  may  lead  to  a  dangerous 
overfitting  of  their  predictive  models:  fundamental  misassumptions  about  the  nature  of 
the  data,  in  turn,  give  rise  to  inaccuracies  in  the  resulting  predictions. 

Reconciling  big  and  small  data.  One  of  the  central  tensions  revealed  in  this  pattern  is, 
therefore,  the  struggle  to  reconcile  big  and  small  data:  how  do  you  represent  the 
specificity  of  individual  points  within  the  big-picture  trends  revealed  by  large  and 
complex  data?  Data  collected  through  social  media  seems  to  promise  a  wealth  of  insight 
on  broad  trends,  social  patterns,  and  consumer  behaviors,  just  as  cell-phone  GPS  data 
offers  unprecedented  tracking  of  commuting,  mobility,  and  navigation  patterns  within 
the  urban  environment.  And  yet  many  researchers  are  struggling  with  the  problem  of 
how  best  to  determine  data  validity  on  such  a  large  scale.  That  is,  how  do  you  most 
effectively  train  algorithms  to  distinguish  an  individual  user,  and  therefore  a  valid  data 
point,  from  unusable  data  generated  by  bots  and  advertisers?  How  do  you  tell  the 
difference  between  a  morning  commuter  and  an  out-of-town  tourist,  if  you  are  using  big 
data  forecasts  to  decide  where  to  construct  new  highways?  Do  these  distinctions  even 
really  matter? 

Obscured  origins  of  epiphenomenon.  As  reliance  on  big  data  leads  to  the 
decontextualization  of  individual  data  points,  so,  too,  does  it  obscure  the  origins  of 
epiphenomenon  arising  from  the  nature  of  the  data  gathering  practice  itself.  For 
instance,  in  the  weeks  leading  up  to  the  recent  Scottish  independence  referendum  vote, 
Amazon  DVD  charts  have  recorded  soaring  sales  of  the  1995  epic  Braveheart,  which 
vaulted  from  1074^*^  to  454*  place. And  yet  while  this  phenomenon  was  short-lived, 
big  data-driven  year-end  tabulations  of  DVD  sale  trends  now  run  the  risk  of  over¬ 
representing  the  film's  general  popularity,  flattening  at  once  the  temporal  transience  of 
such  an  occurrence,  as  well  as  its  significance  as  a  social  artifact  of  a  particular  historico- 
political  event. 


Hooton,  Christopher.  "Scottish  Independence  Referendum  Leads  to  Surge  in  Sales  of  Braveheart."  The 
Independent  [London]  25  Sept.  2014. 
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Big  data  offers  the  tantalizing  possibilities  of  gathering  unprecedented  quantities  of  rich 
information  in  real  time,  increasing  statistical  power  by  orders  of  magnitude  and  providing  new 
depth  and  perspective  to  our  understanding  of  the  operations  of  complex  socio-technical 
systems.  Ultimately  however,  these  few  case  examples  illustrate  that  accessing  big  data  alone  is 
not  enough  to  leverage  its  wide-ranging  potential;  rather,  the  ability  to  extract  meaningful 
forecasting  predictions  from  large  data  sets  relies  on  skillful  analytics  and  the  availability  of 
proper  tools  and  approaches  for  interactively  exploring  and  engaging  with  it  as  well.  In  this 
respect,  an  understanding  of  the  potentials,  and  pitfalls,  of  big  data  demands  a  rigorous 
consideration  of  both  the  capabilities  of  visual  analytic  and  the  nature  of  interactive  modeling 
approaches. 


Visual  Analytics 

Visual  analytics  is  resulting  in  a  transformative  capability,  bridging  human  and  computer 
analysis. 

The  field  of  visual  analytics  has  grown  extensively  over  the  past  decade,  and  there  is  a  large 
body  of  knowledge  on  many  different  aspects.  In  Phase  1,  our  team  made  progress  in  finding 
specific  work  within  this  body  of  knowledge  of  specific  relevance  to  IMCSE,  and  this  will 
continue  in  Phase  2.  Much  of  the  work  on  visual  analytics  focuses  on  natural  data  and  fields  of 
interest  outside  the  scope  of  this  project  (e.g.,  biomedical,  marketing,  etc.).  Uncovering  the 
most  salient  research  findings  is  an  ongoing  effort. 

Uncertainty- Aware  Visual  Analytics.  Correa  et  al.  (2009)^^  discuss  the  growth  of  visual  analytics 
as  an  important  tool  for  gaining  insights  on  large,  complex  data  sets.  The  authors  discuss  the 
problem  of  limitations  on  technology  and  human  power,  making  it  difficult  to  cope  with  the 
growing  scale  and  complexity  of  data,  and  therefore  making  it  is  seldom  possible  to  analyze 
data  in  its  raw  form.  The  data  must  be  transformed  to  a  suitable  representation  in  order  to 
facilitate  discovery  of  interesting  patterns.  However,  the  process  of  transforming  raw  data  to 
abstractions  and  derived  data  is  a  complex  network  of  transformations,  propagating  and 
aggregating  uncertainty.  As  such,  the  authors  believe  when  making  decisions  based  on 
uncertain  data,  it  is  important  to  quantify  and  present  to  the  analyst  both  the  aggregated 
uncertainty  of  the  results  and  the  impact  of  the  sources  of  that  uncertainty,  motivating  their 
work  to  develop  a  framework  for  uncertainty-aware  visual  analytics.  Figure  2  illustrates  the 
process  developed  by  Correa  et  al.,  which  the  authors  describe  as  follows: 

In  general,  visual  analytics  is  the  process  of  transforming  input  data  into  insight. 

A  similar  process  occurs  for  the  uncertainty.  First,  uncertainty  modeling 
generates  a  model  for  source  uncertainty.  As  data  is  transformed,  these 
uncertainties  are  propagated  and  aggregated.  We  obtain  such  estimates  via 
sensitivity  and  error  modeling.  Finally,  the  uncertainty  on  the  derived  data  and  its 


Correa,  Carlos,  Chan,  Yu-Hsuan,  and  Ma,  Kwan-Liu,  "A  Framework  for  Uncertainty-Aware  Visual  Analytics",  IEEE 
Symposium  on  Visual  Analytics  Science  and  Technology,  2009,  p.  51-58 
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sources  are  mapped  to  visual  representations,  which  finally  populate  the  view 
used  by  the  analyst. 


Figure  2.  Uncertainty-aware  visuai  anaiytics  process  (source:  Correa,  et  ai.  2009^'*) 


ANALYST 


Visual  Analytics  Based  Sensemaking.  Vitiello  and  Kalawsky  (2012)^^  discuss  an  approach  that 
integrates  a  visual  analytic  based  workflow  to  the  notion  of  sensemaking.  The  authors  describe 
using  visual  analytics  to  support  systems  thinking  to  make  sense  of  complex  systems 
interactions  and  interrelationships  enabling  rapid  modeling  of  the  systems  of  interest  for 
systems  engineering  design  and  analysis  processes.  They  state  that  sensemaking  evolved  from 
naturalistic  decision  making  research,  as  published  by  Klein  et  al.  (1993)^®.  The  visual  analytic 
based  sensemaking  framework  described  in  their  paper  is  aimed  toward  providing  the  means  to 
rapidly  gain  valuable  insights  into  the  data. 

Work-Centered  Approach  for  Visual  Analytics.  Van  et  al.  (2012)  present  research  on  a  work- 
centered  approach  for  visual  analytics.  The  research  seeks  to  integrate  user-centered  design 
and  data-oriented  data-processing  algorithms  in  order  to  reconcile  human  users'  limited 
capacity  to  process  large  amount  and  rapid  growth  of  information  in  decision  making,  as 
applied  to  tradespace  exploration.  The  authors  state:  "After  a  user  selects  data  of  interest  from 
raw  data,  computational  algorithms  are  applied  to  build  data  models.  The  entire  model  building 
process  is  interactive  to  the  user.  User  has  the  capability  to  control  whether  and  how 
algorithms  run  and  constructs  a  specific  data  model  to  fit  ad-hoc  problems.  Visualization 
provides  an  interface  between  data,  models  and  the  user.  It  displays  both  source  data  and 
computational  results.  It  also  takes  user's  input  and  commands  to  manipulate  on  raw  data  or 
analysis  algorithms". 


Vitiello,  P.  and  Kalawsky,  R.S.,  "Visual  analytics:  A  sensemaking  framework  for  systems  thinking  in  systems 
engineering",  IEEE  International  Systems  Conference  (SysCon),  2012 

Klein,  G.A.,  A  Recognition-Primed  Decision  (RPD)  Model  of  Rapid  Decision  Making,  Decision  making  in  action: 
Modeis  and  methods,  vol  5,  no  4,  pp  138-147,  Dec  1993. 

Van,  X.,  Qiao,  M.,  Li,  J.,  Simpson,  T.W.,  Stump,  G.M.,  Zhang,  X.,  A  Work-Centered  Visual  Analytics  Model  to 
Support  Engineering  Design  with  Interactive  Visualization  and  Data-Mining,  Hawaii  International  Conference  on 
Systems  Science  (HICSS)  2012,  pl845-1854,  Jan  2012 
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Figure  3.  Framework  of  work-centered  visual  analytics  (source  Yan  et  al.^^’ 


Science  of  Interaction.  Our  inquiry  into  visual  analytics  necessitates  looking  into  the  "science  of 
interaction".  Pike  et  al.  discuss  the  interaction  challenges  raised  in  visual  analytics  research, 
and  the  relationship  between  interaction  and  cognition. The  'science  of  interaction',  as 
defined  by  these  authors,  concerns  the  study  of  methods  by  which  humans  create  knowledge 
through  the  manipulation  of  an  interface.  They  state: 

As  visual  analytics  is  concerned  with  the  relationship  between  visual 
displays  and  human  cognition,  merely  developing  novel  visual  metaphors 
is  rarely  sujficient  to  trigger  this  insight  (where  insight  may  be  a  new 
discovery  or  confirmation  or  negation  of  a  prior  belief).  These  visual 
displays  must  be  embedded  in  an  interactive  framework  that  scaffolds  the 
human  knowledge  construction  process  with  the  right  tools  and  methods  to 
support  the  accumulation  of  evidence  and  observations  into  theories  and 
beliefs. 

Seven  key  areas  were  identified  in  this  2009  paper:  ubiquitous,  embodied  interaction;  capturing 
user  intentionality;  knowledge-based  interfaces;  principles  of  design  and  perception; 
collaboration;  interoperability;  and  interaction  evaluation.  Ongoing  research  in  these  areas  will 
be  explored  for  relevant  impact  as  our  research  project  progresses. 

Complex  Systems 

Developing  complex  systems  necessitates  an  approach  to  generate,  manage,  and  analyze 
artificial  data  across  all  aspects  of  system  complexity. 

The  growing  complexity  of  systems  is  well-recognized,  and  investigation  of  system  complexity 
as  related  to  engineered  systems  is  an  active  subject  of  inquiry.  Complexity,  for  instance,  can 


Pike,  W.  A.,  Stasko,  J.,  Chang,  R.,  &  O'connell,  T.,A.  (2009).  The  science  of  interaction.  Information  Visualization, 
8(4),  263-274.  doi:http://dx.doi.org/10.1057/ivs.2009.22 
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relate  to  the  number  of  constituent  and  component  interconnections,  and  to  the  necessary 
rapid  rate  of  information  generation  and  exchange.  It  can  also  relate  to  emergent  behavior  as  a 
result  of  interactions  of  constituent  systems  in  a  system  of  systems. 

Defining  Systems  Complexity.  Many  authors  have  and  continue  to  define  system  complexity. 
Gasper  (2012)^®  discusses  three  bodies  of  work  than  can  be  used  as  a  basis  for  complexity 
definition  in  the  context  of  engineering.  Herbert  Simon  (1962)^°  proposes  that  how  complex  or 
simple  a  structure  is  depends  critically  on  the  way  in  which  we  describe  it.  Simon  proposes  a 
hierarchical  approach  to  complexity,  decomposing  the  system  until  it  can  be  understood. 
Kolmogorov  (1983)^^  definition  of  complexity  asserts  the  more  information  an  object  has,  the 
more  complex  it  is.  Given  the  system  is  the  object,  complexity  can  be  understood  as  related  to 
the  other  objects  that  interact  with  the  system.  The  specification  of  an  object  is  easier  when 
another  object  to  which  this  object  has  a  relation  is  already  specified.  A  third  work  by  Suh 
(2005)^^  discusses  the  idea  of  information  connected  to  the  design  complexity,  proposing  that 
the  violation  of  the  information  axiom,  to  minimize  the  information  content  of  the  design  will 
maximize  the  probability  of  success,  will  result  in  complexity  in  the  system. 

Types  of  System  Complexity.  Structure  and  behavior  are  the  two  aspects  of  complex  systems 
addressed  in  classical  model-based  systems  engineering  Rhodes  and  Ross  (2010)^'^  propose 
five  essential  aspects  for  the  engineering  of  complex  systems:  structural,  behavioral, 
contextual,  temporal,  and  perceptual.  They  argue  that  the  contextual,  temporal  and  perceptual 
aspects  have  been  under-addressed  in  engineering  methods,  and  have  past  and  ongoing 
research  efforts  on  advancing  the  constructs  and  methods  for  contextual,  temporal,  and 
perceptual  aspects.  Response  Systems  Comparison  is  a  resulting  method  to  address  the  five 
aspects^^.  The  method  has  been  applied  in  various  domains  and  for  various  types  of  problems, 
for  example.  Gasper^®  describes  the  application  for  a  conceptual  ship  design  problem. 


Gasper,  H.,  Rhodes,  D.,  Ross,  A.  and  Erikstad,  E.,  "Addressing  complexity  aspects  in  conceptual  ship  design:  a 
systems  engineering  approach",  Journal  of  Ship  Production  and  Design,  Vol.  28,  No.  4,  November  2012,  pp.  1-15 
Simon,  H.,  "The  architecture  of  complexity",  Proceedings  of  the  American  Philosophical  Society,  106,  6,  467- 
482, 1962 

Kolmogorov,  A.  N.,  "Combinatorial  foundations  of  information  theory  and  the  calculus  of  probabilities,  Russian 
Mathematical  Surveys,  38,  4,  27-36, 1983. 

^^Suh,  N.  P.,  "Complexity— theory  and  applications",  Oxford  University  Press,  Oxford,  UK,  2005 
Oliver,  D.,  Kelliher,  T.,  Keegan,  J.,  Engineering  Compiex  Systems  with  Modeis  and  Objects,  NY:  McGraw-Hill,  1997. 
Rhodes,  D.H.  and  Ross,  A.M.,  "Five  aspects  of  engineering  complex  systems:  emerging  constructs  and  methods". 
Proceedings,  4th  Annual  IEEE  Systems  Conference,  April,  San  Diego,  CA,  2010 
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BEIL\TTOR.4L 
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CONTEXTUAL 
related  to  circumstances 
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New  constincts  and  methods  seek  to 
advance  state  of  aiT'.  for  example: 

Epoch  Modeling 

Multi-Epoch  Analysis 

Epoch-Era  Analysis 
Multi-Stakeholder  Negotiations 
Visualization  of  Complex  Data  Sets 

TEMPORAL 

related  to  dimensions 
and  properties  of 
sy  stems  over  time 

PERCEPTU.AL 

related  to  stakeholder 
preferences,  perceptions 
and  cognitive  biases 

Figure  4.  Five  Aspects  of  Complex  Systems^"* 

Human-System  Interaction  Complexity.  The  complexity  of  the  human-system  interaction 
considerations  is  increasingly  important  in  developing  complex  systems.  A  2007  report  of  The 
National  Academies^®  presents  a  discussion  of  the  challenges,  with  research  and  policy 
recommendations.  Many  of  the  points  brought  out  in  this  report  and  in  subsequent  work 
extend  to  understanding  of  complex  systems.  A  number  of  the  recommendations  have 
extensions  to  the  challenges  of  we  see  for  IMCSE,  and  are  beginning  to  be  addressed  through 
research.  Examples  include: 

•  Remote  collaboration  is  difficult  to  participate  in  or  observe  without  proper  remote 
collaboration  tools  enabling  interactivity  of  human  to  human,  and  human  to  model. 

•  Cognitive  and  perceptual  limitations  constrain  the  amount  of  information  that  can  be 
considered  at  a  point  in  time  by  a  single  decision  maker;  multi-sensory  representations 
may  allow  for  some  loosening  of  this  constraint  and  improve  human-model  interaction. 

•  Research  has  increasingly  uncovered  the  important  role  of  context  effects  on  both 
systems  in  use,  design,  and  on  the  decision  makers  themselves.  Facilities  that  can 
represent  and  control  for  these  context  effects  may  uncover  approaches  for  mitigating 
or  taking  advantage  of  these  effects. 


Perceived  and  Descriptive  Complexity.  Project  complexity  has  been  defined  in  many  different 
ways.  In  the  Applications  section  of  this  report,  we  hypothesize  that  perceived  and  descriptive 
complexity  are  correlated  and  constitute  a  tradeoff  between  design-efficiency  and  design- 
robustness  (refer  to  page  71  for  the  detailed  discussion). 


National  Research  Council  (2007),  Human-Systems  Integration  in  the  System  Deveiopment  Process:  A  New  Look, 
Committee  on  Human-System  Design  Support  for  Changing  Technology,  R.W.  Pew  and  A.S.  Mavor,  Eds., 
Committee  on  Human  Factors,  Division  of  Behavioral  and  Social  Sciences  and  Education,  Washington,  DC:  The 
National  Academies  Press. 


27 


Model-Based  Systems  Engineering  (MBSE) 

Model-based  systems  engineering  generates  "artificial  data"  about  our  systems  which  we  use  to 
moke  decisions  that  impact  the  future/continuing  success  of  that  system 

Systems  engineering  is  rapidly  becoming  model-based  in  nature.  It  is  recognized  that  MBSE 
offers  significant  potential^^  but  many  challenges  remain  in  realizing  the  full  potential  of  using 
models  throughout  the  lifecycle  in  numerous  ways.  It  is  recognized  that  the  current  MPTs  are 
inadequate,  and  much  research  and  development  is  ongoing  to  address  this.  A  few  IMCSE- 
related  challenges  we  highlight  are:  integration  of  MPTs,  executable  artifacts,  issues  in  trusting 
models  and  need  for  ontologies. 

Inadequate  MPTs.  The  SERC's  System  2020  -  Strategic  Initiative  Report^^  states:  "Existing 
systems  engineering  tools,  processes,  and  technologies  poorly  support  rapid  design  changes  or 
capability  enhancements  within  acceptable  cost  and  schedule  constraints.  Their  focus  on  point 
solutions  makes  ad-hoc  adaptation  cumbersome  in  theatre.  To  increase  development  efficiency 
and  ensure  flexible  solutions  in  the  field,  systems  engineers  need  powerful,  agile,  interoperable, 
and  scalable  tools  and  techniques".  The  study  concluded  that  "the  purpose,  affordability,  and 
interoperability,  as  well  as  scalability  of  the  computer-aided  design  (CAD)  and  SE  tools  available 
to  DoD  were  weak  with  respect  to  the  complexities  of  future  DoD  missions  and  net-centric 
systems  of  systems."  These  findings  underscore  the  motivation  for  evolving  model-based 
systems  engineering  MPTs  to  enable  users  to  interact  with  models  in  a  more  effective  manner. 

Executable  system  architecture  artifacts.  According  to  the  recent  study  by  the  Systems 
Engineering  Division  of  NDIA^^  "Model  Based  Engineering  (MBE)  is  an  emerging  approach  to 
engineering  that  holds  great  promise  for  addressing  the  increasing  complexity  of  systems,  and 
systems  of  systems,  while  reducing  the  time,  cost,  and  risk  to  develop,  deliver,  and  evolve  these 
systems".  The  study  assessed  the  current  state  of  MBE  and  identified  potential  benefits,  costs 
and  risks  within  the  DoD  acquisition  lifecycle  context. 

According  to  a  recent  SERC  study:^° 

“Modeling  and  Simulation  (M&S)  technology  are  essential  to  understand  the  behavior  of 
the  target  system  and/or  to  evaluate  various  strategies  for  the  operation  of  the  system 
before  it  is  actually  built.  In  many  cases,  simulation  models  reflect  the  design  of  the  final 
system  in  great  detail  and  can  take  the  place  of  architecture  documentation.  In  an  ideal 


Zimmerman,  P.,  A  Review  of  Model-Based  Systems  Engineering  Practices  and  Recommendations  for  Future 
Directions  in  the  Department  of  Defense,  2"'*  Systems  Engineering  in  the  Washington  Metropolitan  Area  (SEDC 
2014)  Conference,  Chantilly,  VA,  April  3,  2014 

SERC-2010-TR-009-1,  Boehm,  B.,  System  2020  -  Strategic  Initiative,  Systems  Engineering  Research  Center,  Final 
Technical  Report,  August  26,  2010. 

NDIA  Systems  Engineering  Division,  Modeling  and  Simulation  Committee,  Final  Report  of  the  Model  Based 
Engineering  (MBE)  Subcommittee,  Feb  10,  2011. 

SERC-2012-TR-024,  zur  Muehlen,M.,  Integration  of  M&S  (Modeling  and  Simulation,  Software  Design  and  DoDAF, 
RT  24,  Systems  Engineering  Research  Center  (SERC),  April  9,  2012 
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scenario,  system  architecture  artifacts  should  be  directly  executable  and  could  be 
leveraged  for  simulation  purposes 

Creating  requisite  process  and  data  models,  as  well  as  use  case  descriptions  can  facilitate  the 
transition  from  requirements  engineering  to  simulation,  and  to  implementation.  The  transition 
from  design  to  implementation,  however,  is  not  seamless.  Differences  in  tool-specific  standard 
implementations  hamper  the  seamless  transition  of  model  information,  increasing  the  burden 
on  the  user. 

Trust  in  Constructed  Models.  A  recent  SERC  study  sponsored  by  the  Naval  Air  Systems 
Command  (NAVAIR),  Introducing  Model-Based  Systems  Engineering:  Transforming  System 
Engineering  through  Model-Based  Systems  Engineering^^  assessed  the  technical  feasibility  of 
creating  and  leveraging  a  more  holistic  MBSE  approach.  The  vision  for  "doing  everything  with 
models"  depends  on  a  common  lexicon  for  MBSE  including  model  levels,  types,  uses,  and 
representations,  and  a  significant  degree  of  automation.  The  sophisticated  model-based 
process  and  enabling  environment  that  are  envisioned  offer  the  potential  for  a  very  powerful 
transformation  of  systems  engineering  through  MBSE.  A  very  significant  challenge  in  realizing 
such  a  vision  is  trust  in  constructed  models^®,  as  we  discuss  below. 

Ontology  for  Human  Systems  Integration.  A  recent  publication  by  Orellana  and  Madni  (2014)^^ 
discusses  the  importance  of  creating  the  ontology  for  human  systems  interaction,  interfaces, 
and  integration.  An  ontology,  according  these  authors,  will  "extend  current  system  modeling 
capabilities  that  will  enable  the  human  element  to  be  analyzed  as  part  of  the  overall  system 
development  process.  As  posed  in  this  paper,  the  role  of  the  human  as  system  operator  is 
evolving  to  that  of  agent,  placing  greater  demands  on  system  architects  and  engineers^^.  The 
ontology,  when  developed,  will  "extend  current  modeling  capabilities  and  allow  the  human 
element  to  be  analyzed  as  part  of  the  overall  system  from  system  conception  to  system 
disposal^^. 


Emerging  Challenges  at  the  Intersection 

Across  the  four  topics  areas,  we've  identified  several  emerging  challenges  at  their  intersection 
with  regard  to  IMCSE.  The  insights  and  techniques  being  developed  in  each  of  the  four  topic 
areas  have  particular  additional  considerations  when  used  to  support  systems  engineering  and 
decision  making.  Each  of  these  challenges  will  now  be  briefly  described.  We  anticipate  using 
these  challenges  to  help  orient  and  motivate  some  of  the  research  activities  within  IMCSE. 


SERC-2014-TR-044,  Blackburn,  M.,  Transforming  Systems  Engineering  through  Model  Based  Systems 
Engineering,  Technical  Report,  Systems  Engineering  Research  Center  (SERC),  March  31,  2014 
Orellana,  D.  and  Madni,  A.,  Human  System  Integration  Ontology:  Enhancing  Model  Based  Systems  Engineering  to 
Evaluate  Human-System  Performance,  Conference  on  Systems  Engineering  Research,  2014,  Procedia  Computer 
Science  28  (2014)  19-25 

Madni,  A.,  Integrating  Humans  with  software  and  Systems,  Technical  Challenges  and  a  Research  Agenda, 
Systems  Engineering  2010, 13  (3),  232-245 
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Tradeoff  of  Models 


Central  to  most  analyses  are  models.  Since  every  model  is  an  abstraction  from  reality,  it  is 
important  for  any  model  user  to  understand  the  implications  of  embedded  assumptions. 
Sensitivity  analysis  is  a  step  often  performed  during  analyses  where  the  stability  of  results  is 
investigated,  as  a  function  of  (often  parametric)  assumptions  (Feuchter  2000)^^.  "Sensitivity 
analyses  should  be  performed  whenever  time  and  resources  allow,  with  an  emphasis  on 
alternatives  that  survived  early  screening  processes"  (OAS  2008)^^.  In  practice,  many  studies 
are  resource  constrained  and  therefore  only  cursory  (if  any)  sensitivity  analysis  is  conducted. 
Since  the  assumptions  in  the  models  impact  the  results  of  those  models,  not  only  are  choices  of 
model  parameters  important  from  a  "within"  model  sensitivity  perspective,  but  also  the  choice 
of  the  model  itself  can  have  large  ramifications  on  the  results.  IMCSE  will  seek  to  address  the 
challenge  of  performing  broad  sensitivity  analysis,  in  terms  of  model  choice,  as  part  of  a  given 
study,  so  that  it  is  not  relegated  to  a  later  activity  that  is  subject  to  omission  when  resources 
are  short. 


Some  preliminary  research  was  done  to  trade  "within  model"  sensitivities  in  value  models, 
investigating  the  potential  for  interaction  in  refining  value  model  parameter  choices  (Ricci  et  al. 
2014).^® 
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Figure  5.  Trust  and  truthfulness  in  value  models  (Ricci  et  al.  2014) 
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IMCSE  will  continue  to  develop  techniques  and  frameworks  for  conducting  trades  on  models 
themselves  and  not  just  within  the  data  generated  by  the  models.  One  example  exploratory 
project  is  described  in  the  "Value  Model  Tradeoff"  section  later  in  this  report. 


Visual  Analytics  of  Artificial  (Model-generated)  Data 

Much  of  the  visual  analytics  literature  highlights  particular  computational  and  user  interaction 
techniques,  or  supporting  infrastructure,  or  applications  to  particular  cases.  Validation  of 
proposed  techniques  and  supporting  infrastructure  typically  hinges  on  matching  user¬ 
generated  insights  and  predictions  to  "ground  truth"  in  the  data.  This  means  that  the  particular 


Feuchter,  C.A.,  "Air  Force  Analyst's  Handbook:  On  Understanding  the  Nature  of  Analysis,"  Office  of  Aerospace 
Studies,  Air  Force  Materiel  Command  (AFMC)  OAS/DR,  Kirtland  AFB,  NM,  www.oas.kirtland.af.mil,  January  2000. 

Office  of  Aerospace  Studies  (OAS),  "Analysis  of  Alternatives  (AoA)  Handbook:  A  Practical  Guide  to  Analyses  of 
Alternatives,"  Air  Force  Materiel  Command  (AFMC)  OAS/A9,  Kirtland  AFB,  NM,  www.oas.kirtland.af.mil,  July  2008. 

Ricci,  N.,  Schaffner,  M.A.,  Ross,  A.M.,  Rhodes,  D.H.,  and  Fitzgerald,  M.E.,  "Exploring  Stakeholder  Value  Models 
Via  Interactive  Visualization,"  12th  Conference  on  Systems  Engineering  Research,  Redondo  Beach,  CA,  March  2014 
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data  set  being  explored  via  VA  tends  to  be  rooted  in  natural  (i.e.  empirical)  data,  where  "ground 
truth"  has  meaning.  Once  this  is  the  case,  visualizations  for  pattern  matching  by  humans  is 
more  likely  to  be  uncovering  actual  patterns  in  the  data,  rather  than  artifacts.  (Artifacts  may 
still  exist  due  to  data  errors,  sensor  errors,  or  data  abstraction  and  aggregation  effects,  for 
example.  However,  these  effects  can  be  managed  if  a  valid  (i.e.  "true")  dataset  is  available.)  An 
example,  of  this  dynamic  is  displayed  in  the  MIT  Big  Data  Challenge.  In  this  contest,  a  large  data 
set  of  historical  taxi  data  and  other  related  data  sets  are  provided  to  competitors.  Competitors 
must  develop  predictive  models  of  number  of  taxi  trips  as  a  function  of  location.  The  scoring  of 
the  predictive  models  "will  be  computed  as  the  root-mean-squared  error  of  [the]  predictions 
against  the  ground  truth."^^ 

Since  the  goal  of  visual  analytics  is  to  generate  insights  into  relationships  and  patterns  in  the 
data,  the  existence  of  potentially  confounding  artifacts  in  the  data  makes  it  especially 
challenging  when  ground  truth  is  no  longer  available.  This  is  essentially  the  difference  between 
exploratory  modeling  and  consolidative  modeling  (Bankes  1993)^®.  Consolidative  modelling 
includes  "techniques  in  which  known  facts  are  consolidated  into  a  single  model"  in  order  to 
generate  explanatory  relationships  of  existing  data  (Kwakkel  and  Pruyt  2012).^®  While  in 
exploratory  modelling,  the  intention  is  to  "generate  artificial  data"  that  "can  inform  modelers 
and  decision  makers  of  the  ramifications  of  various  sets  of  assumptions,  as  well  as  provide 
consistent  communication"  (Schaffner  2014)^°. 

In  IMCSE,  models  will  tend  to  be  of  exploratory  nature  and  therefore  additional  considerations 
must  be  taken  into  account  when  generating  and  visualizing  the  data  in  order  to  properly 
interpret  the  results. 

Perceptual  and  Cognitive  Considerations  in  Human-Model  Interaction 

In  considering  the  form  of  visual  analytics  to  represent  big  data,  and  the  structure  of  model- 
based  approaches  to  forecasting  the  evolving  complexities  of  large-scale  system,  it  is  crucial  to 
also  consider  the  perceptual  and  cognitive  capabilities  of  human  beings  at  the  center  of  these 
exploratory  efforts. 

There  are  many  considerations  in  human-model  interaction,  and  relevant  research  crosses 
multiple  disciplines.  For  example,  recent  neurocognitive  investigations  offer  some  insight  into 
three  behavioral  phenomena  related  to  decision-making  which  may  provide  a  structural 
framework  for  guiding  these  considerations:  1)  the  behavioral  over-reliance  on  cognitive  biases 
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in  choice  behavior;  2)  the  tendency  towards  ambiguity  aversion;  and  3)  the  limitations  of 
affective  forecasting  when  making  projections  about  future  needs  and  desires. 

Behavioral  Over-Reliance  on  Cognitive  Biases  in  Choice  Behavior.  Broadly  speaking, 
cognitive  biases  arise  from  a  maladaptive  overreliance  on  heuristics,  a  series  of  cognitive 
'short-cuts'  human  beings  recruit  to  reduce  the  complexity  of  day-to-day  decisions,  and 
thereby  decrease  cumulative  cognitive  loading. Heuristics  allow  individuals  to 
extrapolate  from  the  consequences  of  previous  decision-making  events  to  inform  future 
choice  behavior.  Yet  when  individuals  become  overly  reliant  on  such  strategies,  at  the 
exclusion  of  considering  novel,  situation-specific  information,  they  become  biased,  and 
often  demonstrate  impaired  decision-making  abilities.  However,  research  has  suggested 
that  cycles  of  cognitive  bias  can  be  broken  through  training  and  self-monitoring,  and 
that  the  effective  visual  presentation  of  information  may  reduce  a  reliance  on  biased 
strategies  and  promote  thoughtful  consideration  of  salient  data  points,  leading  in  turn 
to  better  and  more  informed  choice  patterns. 

Human  Tendency  towards  Ambiguity  Aversion.  A  second  important  neurocognitive 
consideration  is  the  processing  of  information  regarding  risk  and  ambiguity.  Recent 
studies  investigating  the  neural  correlates  of  decision-making  have  shown  distinct 
patterns  of  brain  activation  in  response  to  uncertainty Behaviorally,  it  has  been  well 
established  that  a  risky  option  of  known  probability— even  when  the  odds  are  poor— is 
often  favored  over  one  where  the  decider  is  ignorant  of  the  precise  degree  of  risk,  a 
phenomenon  termed  'ambiguity  aversion.'  Broadly  speaking,  these  findings  point  to  a 
general  human  intolerance  for  ambiguity  and  a  preference  for  information  seeking.  In 
this  regard,  one  of  the  advantages  of  big  data  driven,  model-based  forecasts  is  to  reduce 
ambiguity  by  extrapolating  future  patterns  from  previously  observed  occurrences, 
thereby  generating  new  and  useful  information  for  decision-makers. 

Limitations  of  Affective  Forecasting  When  Making  Projections.  Finally,  efforts  in 
experimental  psychology  to  probe  human  abilities  to  'affectively  forecast'— that  is,  to 
make  accurate  projections  about  their  future  wants,  desires,  and  emotional  states— 
have  revealed  that,  on  average,  people  are  in  fact  quite  inaccurate  in  determining  the 
emotional  consequences  of  future  events,  often  overestimating  the  amount  of  future 
satisfaction  a  given  set  of  events  will  bring  them.^^  To  this  end,  model-based  forecasts 


Tversky,  A.,  &  Kahneman,  D.  (1974).  Judgement  under  uncertainty:  Heuristics  and  biases.  Sciences,  185(4157): 
1124-1131. 

Brand,  M.,  Labudda,  K.,  &  Markowitsch,  H.  J.  (2006).  Neuropsychological  correlates  of  decisionmaking  in 
ambiguous  and  risky  situations.  Neural  Networks,  19(8),  1266-1276. 
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which  project  future  states  by  examining  current  and  past  trends  may  prove  essential  to 
aiding  decision-making  about  the  future  which  may  otherwise  be  tainted  by  inaccurate 
assumptions  of  impending  affective  state. 

When  taken  together,  these  three  streams  of  neurocognitive  research  highlight  the  ways  in 
which  a  fundamental  understanding  of  people's  perceptual  and  cognitive  capabilities  allows  for 
rich,  human-centered  design  in  the  presentation  of  visual  analytic  displays  and  engaging, 
interactive  models  which  facilitate  exploration  and  discovery.  At  the  same  time,  work  from 
these  fields  also  illuminates  ways  in  which  visual  analytic  approaches  and  model-based 
projections  may  serve  as  effective  aids  for  complex,  real-world  decision-making. 
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Fundamentals 


The  fundamentals  thrust  presently  includes  three  areas:  Interactive  Epoch-Era  Analysis  project, 
Value-Model  Tradeoff,  and  Supporting  MPTs. 


Interactive  Epoch-Era  Analysis 

Epoch-Era  Analysis  is  a  framework  that  supports  narrative  and  computational  scenario  planning 
and  analysis  for  both  short  run  and  long  run  futures.  This  project  is  performing  exploratory 
development  of  interactive  Epoch-Era  Analysis,  including  human  interface  and  reasoning 
considerations  for  epoch  and  era  characterizations,  os  well  as  single  and  multi-  epoch/era 
analyses. 


Background 

Epoch-Era  Analysis  (EEA)  is  an  approach  designed  to  clarify  the  effects  of  changing  contexts 
over  time  on  the  perceived  value  of  a  system  in  a  structured  way  (Ross  2006''®,  Ross  and  Rhodes 
2008''^).  The  base  unit  of  time  in  EEA  is  the  epoch,  which  is  defined  as  a  time  period  of  fixed 
needs  and  context  in  which  the  system  exists.  Epochs  are  represented  using  a  set  of  epoch 
variables,  which  can  take  on  either  continuous  or  discrete  values.  These  variables  can  be  used 
to  represent  any  exogenous  uncertainty  that  might  have  an  effect  on  the  usage  and  perceived 
value  of  the  system;  weather  conditions,  political  scenarios,  financial  situations,  operational 
plans,  and  the  availability  of  other  technologies  are  all  potential  epoch  variables.  Appropriate 
epoch  variables  for  an  analysis  include  key  (i.e.,  impactful)  exogenous  uncertainty  factors  that 
will  affect  the  perceived  success  of  the  system.  A  large  set  of  epochs,  differentiated  using 
different  enumerated  levels  of  these  variables,  can  then  be  assembled  into  eras,  ordered 
sequences  of  epochs  creating  a  description  of  a  potential  progression  of  contexts  and  needs 
over  time.  This  approach  provides  an  intuitive  basis  upon  which  to  perform  analysis  of  value 
delivery  over  time  for  systems  under  the  effects  of  changing  circumstances  and  operating 
conditions,  an  important  step  to  take  when  evaluating  large-scale  engineering  systems  with 
long  lifespans. 

Encapsulating  potential  short  run  uncertainty  (i.e.  what  epoch  will  my  system  experience  next?) 
and  long  run  uncertainty  (i.e.  what  potential  sequences  of  epochs  will  my  system  experience  in 
the  future?)  allows  analysts  and  decision  makers  to  develop  dynamic  strategies  that  can  enable 
resilient  systems.  Key  challenges  in  application  of  EEA  up  to  this  point  involve  eliciting  a 
potentially  large  number  of  potential  relevant  epochs  and  eras,  conducting  analysis  across 
these  epochs  and  eras,  and  extracting  useful  and  actionable  information  from  the  analyses. 
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Schaffner  (2014)"^^  showed  that  the  number  of  potential  eras  to  consider  grows  very  quickly, 
becoming  computationally  infeasible. 

Potential  era  space  =  where  L  is  the  number  of  levels  of  epoch  variable  i,  v  is  the 

number  of  epoch  variables,  and  n  is  the  number  of  epochs  in  a  given  era.  As  an  example,  a 
model  of  5  epoch  variables  with  3  levels  each  and  an  era  of  10  epochs,  would  result  in  3^°,  or 
~10^^  different  eras. 

This  means  that  for  many  problem  formulations  it  is  not  feasible  to  evaluate  systems  across  all 
or  even  a  large  fraction  of  potential  eras"^®.  As  described  earlier  in  the  four  pillars  of  IMCSE,  big 
data  and  visual  analytics  both  have  led  to  techniques  that  could  be  leveraged  to  mitigate  these 
challenges.  It  is  hypothesized  in  this  research  that  adding  interactivity  to  EEA  will 
fundamentally  enable  new  capabilities  and  insights  to  be  derived  from  an  EEA,  resulting  in 
superior  dynamic  strategies  for  resilient  systems.  In  particular,  we  have  three  informal 
hypotheses  regarding  interactive  EEA  (iEEA): 

1.  iEEA  will  enable  the  elicitation  of  more  broad/complete  set  of  possible  epochs: 

a.  Infrastructure  that  enables  iEEA  could  include  databases  of  epoch  variables, 
which  could  be  leveraged  in  future  iEEA  studies 

b.  Explicit  implementations  in  an  interface  will  provide  repeatable  and  more 
understandable  elicitation  experiences,  resulting  in  more  epoch  variables 

2.  iEEA,  through  human-in-the-loop  implementation,  will  help  to  intelligently  limit  the 
potentially  unbounded  growth  in  the  potential  epoch/era  space: 

a.  Using  visual  analytic  techniques  such  as  filtering,  binning,  pattern  matching, 
human  with  computer  iEEA  can  be  used  to  effectively  manage  multi-epoch  and 
multi-era  analysis  scale  growth 

3.  iEEA  will  enable  the  development  of  superior  intuition,  buy-in,  and  insight  generation 
for  decision  making: 

a.  Through  getting  people  to  "experience"  (i.e.  "see"  and  "interact  with")  epochs 
and  eras,  they  will  better  understand  and  accept  the  impact  of  context  and 
needs  changes  on  systems  and  therefore  how  resilience  can  be  better  achieved 

Earlier  work  demonstrated  promise  for  such  capability  and  insight  improvement  when 
interactivity  is  added  to  tradespace  exploration.  Ross  et  al.  (2010)“  introduces  a  method, 
applied  to  two  aerospace  cases  in  order  to  explore  the  potential  for  interactive  tradespace 
exploration  to  support  stakeholder  negotiations.  Preliminary  results  indicate  the  method  to  be 


Schaffner,  M.A.,  Designing  Systems  for  Many  Possible  Futures:  The  RSC-based  Method  for  Affordable  Concept 
Selection  (RMACS),  with  Multi-Era  Analysis,  Master  of  Science  Thesis,  Aeronautics  and  Astronautics,  MIT,  June 
2014. 

Schaffner  2014  suggested  several  possible  mitigations  to  this  problem,  including  human  in  the  loop  era  tree 
pruning,  which  will  be  investigated  in  this  research  project. 

“  Ross,  A.M.,  McManus,  H.L.,  Rhodes,  D.H.,  and  Hastings,  D.E.,  "A  Role  for  Interactive  Tradespace  Exploration  in 
Multi-Stakeholder  Negotiations,"  AIAA  Space  2010,  Anaheim,  CA,  September  2010 
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a  rapid  and  beneficial  technique,  which  generated  compromise  alternatives,  guided  the 
elicitation  of  previously  unarticulated  information,  and  resulted  in  increased  confidence  and 
solution  buy-in  of  participating  stakeholders.  Interactive  tradespace  exploration  analyses 
allowed  negotiation  processes  to  proceed  quickly.  Proposed  compromises  can  be  assessed  by 
each  stakeholder  in  real  time,  and  what  the  stakeholder  is  gaining  or  losing  in  the  compromise 
is  immediately  visible.  An  open  area  of  research  is  to  incorporate  Epoch-Era  Analysis  into  the 
interactive  tradespace  exploration. 

Introduction 

The  development  of  engineered  resilient  systems  (ERS)  was  identified  as  a  science  and 
technology  (S&T)  priority  for  the  DoD  by  the  Secretary  of  Defense  in  April  2011.  Since  that  time 
several  researchers  and  practitioners  have  begun  to  develop  methods,  techniques  and  tools  to 
assist  designers  in  the  early  system  concept  selection  phase.  Many  of  the  techniques  in 
development  require  analysis  of  vast  amounts  of  data  to  quantify  the  effectiveness  of  large 
numbers  of  actionable  alternatives  across  large  numbers  of  possible  futures  in  order  to  select 
the  best  possible  decision.  Recognizing  that  some  human-in  the-loop  techniques  that  are 
being  pioneered  in  studies  of  visual  analytics  and  big  data  analysis  may  assist  in  solving  this 
problem,  this  research  seeks  to  leverage  and  expand  upon  those  techniques.  The  challenge  this 
research  seeks  to  address  can  be  described  as:  "how  can  one  balance  System,  Context,  and 
Expectations  over  time,  during  engineering  design,  evaluation  and  selection,  given  human 
cognitive  and  perceptual  limitations?"  (Ross  2014)^^ 

The  development  of  complex  engineering  systems  using  traditional  engineering  design 
techniques  can  lead  to  point  designs  optimized  for  a  fixed  operating  context  or  set  of 
stakeholder  needs.  This  can  reduce  system  performance  if  future  uncertainty  resolves  in  a  way 
other  than  predicted.  This  is  especially  true  if  the  system  is  not  resilient  or  robust  to  change. 
As  an  example,  consider  modern  spacecraft,  which  have  long  development  timelines  of  5  to  10 
years  or  more  that  makes  them  susceptible  to  changes  in  mission  and  technology  before  they 
even  reach  orbit.  They  must  also  have  a  significant  amount  of  redundancy  built  in  because  a 
replacement  system  could  take  years  to  develop  and  launch  if  they  fail.  Reducing  such 
susceptibilities  to  changes  in  context  was  a  key  goal  of  DARPA's  System  F6  program.  A  shift  in 
stakeholder  needs  for  which  the  system  is  not  resilient  can  also  limit  its  value  delivery.  A 
noteworthy  example  is  the  Iridium  satellite  constellation  that  suffered  from  a  shift  in  the 
consumer  market  to  land-based  cellular  towers  before  it  reached  initial  operating  capability 
(IOC)  (Curry  2014)”. 

The  definition  of  what  is  or  is  not  a  resilient  system  is  not  universally  agreed  upon.  One 
definition  is  that  a  resilient  system  has  "the  ability  to  circumvent,  survive,  and  recover  from 
failures  to  ultimately  achieve  mission  priorities  even  in  the  presence  of  environmental 


Ross,  A.M.,  "Interactive  Model-Centric  Systems  Engineering,"  Presentation  to  the  5th  Annual  SERC  Sponsor 
Research  Review,  Georgetown  University,  Washington,  DC,  February  2014. 

Curry,  M.,  "Presentation:  Application  of  Epoch  Era  Analysis  to  the  Design  of  Engineered  Resilient  System",  17th 
NDIA  Systems  Engineering  Conference,  Springfield,  VA,  October,  2014. 
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uncertainty"  (Madni  2012)^^.  Yet  another  definition  of  resilience  (called  system  "survivability" 
elsewhere,  adding  to  semantic  confusion)  is  "the  ability  of  a  system  to  minimize  the  impact  of  a 
finite  duration  disturbance  on  value  delivery,  achieved  through  either  (1)  the  reduction  of  the 
likelihood  or  magnitude  of  a  disturbance;  (2)  the  satisfaction  of  a  minimally  acceptable  level  of 
value  delivery  during  and  after  a  finite  disturbance  or;  (3)  timely  recovery  from  a  disturbance 
event"  (Richards  et  al.  2007)^^. 

More  recent  work  has  generalized  this  concept  into  something  called  value  sustainment 
(Beesemyer  2012^^).  Value  sustainment  is  defined  as  "the  ability  to  maintain  value  delivery  in 
spite  of  epoch  shifts  or  disturbances."  Figure  6  below  summarizes  this  concept  and  reflects  how 
we  will  consider  notions  of  resilience  in  this  research  effort.  In  this  figure,  the  nominal  value 
delivered  by  a  system  is  (potentially)  impacted  by  a  perturbation  (characterized  as  either  a 
disturbance  or  a  shift).  A  disturbance  is  a  short  duration,  likely  to  revert  imposed  change  on  the 
design,  context,  or  needs  for  a  system,  while  a  shift  is  a  long  duration,  unlikely  to  revert 
imposed  change  on  the  design,  context,  or  needs  for  a  system.  A  "resilient"  system  is  one  that 
either  is  not  impacted,  or  maintains  value  above  the  indicated  threshold,  and  restores  that 
value  delivery  to  a  higher  acceptable  level  after  a  threshold  period  of  time.  What  is  common  to 
most  of  the  definitions  suggested  for  resilient  systems  is  an  acknowledgement  that  complex 
systems  must  be  designed  to  continue  to  deliver  sustained  value  to  their  stakeholders  even  if 
uncertainty  exists  about  the  way  a  system  will  be  required  to  operate  in  the  future. 

“Long”  duration  perturbation  “Short”  duration  perturbation 

(unlikely  to  revert)  (likely  to  revert) 
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Figure  6.  Temporal  representation  of  system  over  time  and  two  perturbation  types:  epoch  shifts  and 
disturbances 


EEA  AND  Data  Challenges 

Traditional  tradespace  exploration  and  multidisciplinary  design  optimization  techniques 
typically  assume  as  fixed  the  needs  of  the  stakeholders,  the  context  in  which  a  system  will  be 
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operated  and  the  future  state  of  the  system  itself.  To  design  resilient  systems  we  must 
consider  situations  in  which  these  can  all  vary  with  time.  One  framework  for  evaluating  such 
possibilities  is  Epoch  Era  Analysis  (Ross  2006)“.  EEA  conceptualizes  the  effects  of  time  and 
changing  context  on  a  system  by  modeling  combinations  of  future  context  and  stakeholder 
needs  on  perceived  system  value  (Ross  and  Rhodes  2008“;  Fitzgerald  et  al.  201l“;  Schaffner  et 
al.  2013“).  A  time  period  over  which  the  stakeholder  needs  and  the  context  in  which  the 
system  must  operate  are  fixed  is  referred  to  as  an  epoch.  A  series  of  epochs  can  be  strung 
together  to  form  eras  that  can  be  used  to  model  the  long-run  value  delivery  of  a  system  and 
take  into  account  temporal  path  dependencies  between  epochs.  Such  eras  can  be  generated 
through  narrative  (i.e.  story-driven)  or  computational  means  (i.e.  algorithm-generated) 
enabling  consideration  of  a  broader  set  of  possible  short  and  long  run  scenarios  than  commonly 
considered  using  traditional  scenario  planning  techniques  (Roberts  et  al.  2009)“. 

Broadly  speaking,  EEA  be  described  as  the  following  activities  (roughly  sequential  and  depicted 
in  Figure  7): 

0.  Problem  Definition:  identify  decision  to  be  made,  relevant  constraints,  stakeholders,  and 
potential  contexts 

1.  Design  Formulation:  generate  potential  design  alternatives  to  be  evaluated  in  the  analysis; 
can  be  generated  via  inheritance,  creative  brainstorming,  value-driven  methods,  or 
other  means;  identify  preliminary  criteria  for  their  evaluation. 

2a.  Epoch  Characterization:  identify  key  exogenous  uncertainties  and  parameterize  via  epoch 
variables;  can  be  accomplished  via  era  deconstruction  or  proposing  possible  short  run 
scenarios. 

2b.  Era  Construction:  generate  various  long  term  descriptions  of  possible  futures  via  epoch 
sequencing,  or  proposing  long  run  scenarios  (e.g.  via  narrative  or  computational 
means). 

3.  Design-Epoch-Era  Evaluations:  develop  and  execute  appropriate  models  that  can  evaluate 
designs  in  epochs  in  eras. 

4a.  Single  Epoch  Analyses:  conduct  analyses  of  the  designs  within  particular  epochs, 
determining  performance  and  cost  of  alternatives  and  difficulty  of  achieving  success 
within  particular  periods  of  fixed  context  and  needs. 
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Changeability  in  System  Design,"  9th  Conference  on  Systems  Engineering  Research,  Los  Angeles,  CA,  April  2011. 
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4b.  Single  Era  Analyses:  conduct  analyses  within  particular  eras  to  determine  the  impact  of 
time-dependent  effects  on  system  success,  along  with  cumulative  path-dependence  on 
the  system  over  time. 

5a.  Multi-Epoch  Analysis:  conduct  analysis  across  multiple  (or  all)  epochs  to  determine 
sensitivities  of  designs  to  epochs;  gives  insight  into  short  run  value  of  active  and  passive 
strategies  for  system  resilience. 

5b.  Multi-Era  Analysis:  conduct  analysis  across  multiple  (or  all)  eras  to  determine  sensitivities 
of  designs  to  eras  and  patterns  of  path  dependence;  gives  insights  into  long  run  value 
of  active  and  passive  strategies  for  system  resilience. 


Figure  7.  Activities  in  Epoch-Era  Analysis 


Figure  8  illustrates  the  era-tree  approach  to  era  construction  via  paths  through  the  epoch 
space.  Each  epoch  is  defined  as  a  particular  context-need  pair  and  duration. 


Figure  8.  Eras  generated  as  path  through  possible  epoch  space  (from  Ross  et  al.  2008^^) 
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Schaffner  (2014)  introduces  useful  terms  when  constructing  eras.  A  frame  is  a  particular  slot 
within  an  era  that  consists  of  an  epoch  and  a  duration.  This  allows  an  EEA  user  to  specify  era  of 
varying  number  of  slots  in  a  less  ambiguous  manner.  For  example,  a  5  frame  era  consists  of  5 
slots,  each  with  a  particular  epoch  and  duration.  The  same  epoch  could  appear  in  more  than 
one  frame.  A  second  useful  concept  is  that  of  a  clip,  which  is  a  subset  of  a  full  era,  comprised  of 
an  arbitrarily  small  number  of  frames.  Using  this  nomenclature,  one  can  speak  of  3-frame  clips, 
for  example,  which  might  appear  in  multiple  different  eras.  When  looking  for  patterns,  such  a 
unit  of  analysis  may  be  useful. 


Epoch 

Figure  9.  Epochs  as  Alternative  "Point"  Futures  (I)  and  Multi-Epoch  Analysis  (r) 

Figure  9  illustrates  epochs  as  alternative  (point)  futures,  and  multi-epoch  analysis  as  a  cross¬ 
epoch  activity  looking  for  designs  that  perform  well  across  the  alternative  future  space. 

A  practical  challenge  in  implementations  of  EEA  is  the  large  amount  of  data  that  may  need  to 
be  evaluated  in  order  to  thoroughly  characterize  possible  system  alternatives  and  their 
potential  for  value  sustainment  across  a  wide  variety  of  futures.  In  EEA,  large  numbers  of 
possible  alternatives  may  need  to  be  generated  by  evaluating  many  possible  combinations  of 
design  variables  using  performance  models.  Evaluating  large  numbers  of  designs  is  common 
tradespace  exploration  challenge,  especially  if  performance  models  take  a  nontrivial  amount  of 
time  and  resources  to  evaluate.  The  problem  is  further  exacerbated  by  the  need  to  evaluate 
each  possible  design  alternative  in  each  possible  future  context  and  across  many  future 
stakeholder  value  models.  This  can  quickly  turn  into  an  intractable  problem  with  millions  or 
billions  (or  more)  of  data  points  that  must  be  analyzed  as  part  of  the  decision  problem. 
Techniques  from  visual  analytics  may  allow  a  human-in-the-loop  to  more  quickly  identify 
patterns  in  the  data  and  filter,  sort  and  aggregate  data  more  efficiently  than  "canned"  or 
automated  algorithms,  enabling  a  more  effective  tradeoff  of  evaluation  "completeness"  versus 
insights  gained. 

As  previously  noted,  applying  EEA  to  some  system  design  problems  may  lead  to  vast  amounts 
of  data.  New  techniques  that  allow  a  decision  maker  to  interact  with  their  data  to  gain  insights 
may  improve  the  systems  engineering  decision  making  process.  Liu  et  al.  (2013)“  and  Heer  and 
Shneiderman  (2012)“  point  out  that  "interaction  is  essential  to  exploratory  visual  analysis",  but 
their  work  primarily  focuses  on  visualization.  Note  that  interaction,  as  used  here,  is  not 
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intended  to  be  strictly  limited  to  the  data  visualization  component,  but  also  the  interfaces, 
processes,  and  methods  that  allow  a  user  to  gain  insights  from  their  data.  Interfaces  may 
require  use  of  sensory  stimuli  other  than  visual-only,  including  touch  and/or  sound.  Processes 
could  include  custom  workflows  such  as  those  described  in  Sitterle  et  al.  (2014)“.  Methods  for 
sorting  and  filtering  data  may  include,  but  are  not  limited  to,  interactive  brushing  and  linking  of 
multiple  simultaneous  visual  displays. 

The  problems  that  may  arise  when  scaling  up  to  larger  decision  problems  with  traditional  EEA 
could  be  placed  into  four  categories: 

1.  Data  size  increases  which  creates  a  storage  and  data  transmission  problem. 

2.  Data  size  increase  also  creates  a  separate  problem  related  to  cross-filtering  across  large 
numbers  of  data  dimensions.  Human  cognitive  limitations  make  comprehension  of 
high-dimensional  data  difficult  so  datasets  must  be  "sliced"  or  cross  tabulated  across 
dimensions  before  rendering  them  as  ID,  2D  or  3D  visualizations. 

3.  Larger  data  sets  require  increased  amounts  of  processing  time  to  manipulate. 

4.  Rendering  problems  arise  when  large  amounts  of  data  must  be  visualized 
simultaneously. 


A  Framework  for  Interactive  Epoch-Era  Analysis 

The  current  vision  of  Interactive  Epoch-Era  Analysis  (iEEA)  leverages  humans-in-the-loop  as  well 
as  supporting  infrastructure  in  order  to  manage  challengers  associated  with  the  large  amounts 
of  data  potentially  generated  in  a  study,  as  well  as  sense  making  of  the  results.  Figure  10  below 
illustrates  three  insertion  points  for  interactivity  to  directly  address  the  three  hypotheses 
outlined  earlier  (improved  elicitation,  improved  analyses,  improved  decision-making). 


“  Sitterle,  V.,  Curry,  M.,  Ender,  T.,  Freeman,  D.,  "Integrated  Toolset  and  Workflow  for  Tradespace  Analytics  in 
Systems,"  INCOSE  International  Symposium,  Las  Vegas,  NV,  2014. 
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Figure  10.  Interactive  Epoch-Era  Analysis  leverages  humans-in-the-loop  and  supporting  infrastructure 

iEEA  seeks  to  combine  several  techniques  to  address  the  problems  described  above.  These 
problems  can  be  resolved  or  mitigated  by  using  a  hybrid  approach  that  combines  techniques 
from  parallel  computing,  online  analytical  processing  and  visual  analytics.  Problems  with 
rendering  and  the  scalability  of  visualizations  and  other  encoded  visual  information  can  be 
improved  upon  using  techniques  that  do  not  require  every  single  data  point  to  be  drawn.  Liu 
points  out  that,  "Perceptual  and  interactive  scalability  should  be  limited  by  the  chosen 
resolution  of  the  visualized  data,  not  the  number  of  records,"  and  summarizes  several 
techniques  past  researchers  have  applied  to  reduce  the  pixel  density  of  visualizations  (Liu  et  al. 
2013)®^ 

1.  Data  reduction  through  filtering 

2.  Data  reduction  through  sampling 

3.  Binned  aggregation 

4.  Model-fitting 

Implementation  of  iEEA  demonstration  tools  and  methods  may  need  to  draw  on  a  combination 
of  the  techniques  described  above.  This  means  that  iEEA  needs  to  take  into  account  the 
practicality  of  representing  large  amounts  of  data  effectively  view  scarce  communication 


Liu,  Z.  Jiang,  B.,  Heer,  J.,  "imMens:  Real-time  Visual  Querying  of  Big  Data,"  Eurographics  Conference  on 
Visualization  (EuroVis),  2013. 
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resources  (e.g.  limited  spatial  or  temporal  resolutions  due  to  hardware  or  software  constraints) 
(Keim  2000®®).  Given  the  volume  and  complexity  of  the  data  that  will  need  to  be  analyzed,  iEEA 
should  allow  a  decision  maker  to  manipulate  their  viewpoint  into  the  data  in  real-time.  Their 
viewpoint  can  change  both  in  level  of  abstraction  and  information  type  as  shown  in  Figure  11 
below.  One  possible  analogy  is  to  compare  iEEA  techniques  to  those  used  to  interact  with  web- 
based  map  applications  such  as  Google  Maps.  When  zooming  in  and  out  (level  of  abstraction) 
of  the  default  road  map  the  scale  of  information  displayed  to  the  user  changes  automatically  as 
roads,  city  names  and  other  feature  are  enabled  or  hidden.  Attempting  to  view  all  available 
data  from  the  top  zoom  scale  could  create  a  cluttered  visualization  that  is  not  constructive  or 
helpful  for  allowing  the  decision  maker  to  reach  a  conclusion. 

It  is  also  envisioned  that  with  iEEA  the  user  could  also  adjust  the  layer  of  information  they  are 
looking  at  (information  type)  to  make  their  decision.  Depending  on  the  type  of  decision  they 
are  making  they  may  need  to  switch  across  various  types  of  displays  or  variables  at  the  same 
zoom  level.  The  information  they  need  may  be  a  satellite  image,  terrain  map,  traffic  map, 
public  transit  map  or  some  hybrid  combination.  Clearly  the  visualizations  needed  for  the  types 
of  decisions  iEEA  supports  will  likely  not  be  maps,  but  the  analogy  of  how  users  will  make  a 
large  data  set  manageable  for  making  decisions  through  interactive  techniques  is  appropriate. 


Level  of 
Abstraction 


Figure  11.  Level  of  abstraction  versus  information  type  for  map  data  (individual  images  from  maps.google.com) 


Keim,  D.:  Designing  pixel-oriented  visualization  techniques:  Theory  and  applications.  IEEE  TVCG  6,  1  (2000),  SO¬ 
TS. 
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Preliminary  Work  Exploring  Techniques  from  VA  and  Big  Data  with  Applicability  to  EEA 

In  order  to  develop  an  effective  framework  for  interactive  Epoch-Era  Analysis,  this  project  has 
pursued  an  initial  set  of  exploratory  prototype  mini-projects.  These  have  focused  so  far  on 
leveraging  techniques  from  big  data  as  applied  to  EEA  in  order  to  manage  the  data  scale 
challenge  potentially  faced  during  iEEA.  These  prototype  accomplishments  include: 

1.  Example  of  massive  parallel  processing  that  takes  advantage  of  Amazon  cloud 
computing  services  to  generate  very  large  design/epoch  spaces. 

2.  Example  web-based  tools  that  integrated  interactive  D3-based®^  graphics  with  a 
database  driven  backend.  This  prototype  showed  that  we  could  create  an  interactive 
visual  interface  that  was  driven  by  legacy  databases  such  as  the  Space  Tug®®  database 
(Figure  12,  Figure  13,  and  Figure  14).  It  also  demonstrates  techniques  for  more  easily 
appending  or  culling  a  database  from  within  the  interactive  interface  rather  than  relying 
on  a  3rd  party  piece  of  software. 

3.  Recent  example  that  extends  the  above  example  to  use  online  analytical  processing 
(OLAP)  techniques  to  enable  a  decision  maker  to  more  rapidly  filter  their  data  to  identify 
patterns  in  high-dimensional  data. 

4.  A  final  example  (hexagonal  binning)  to  show  that  we  can  extend  the  above  prototype 
system  to  easily  be  capable  of  handling  1,000,000  alternatives  that  can  be  conveyed 
quickly  and  effectively  to  a  decision  maker  in  an  interactive  way  (Figure  15). 


Figure  12  is  a  screenshot  of  a  prototype  tool  implemented  in  D3  demonstrating  coordinated 
views  using  online  analytical  processing  to  "slice"  across  dimensions  or  perform  "drill-downs" 


D3.js  is  a  JavaScript  library  for  enabling  interactive  web  documents  based  on  data  (http://www.d3is.org) 
McManus,  H.,  and  Schuman,  T.,  "Understanding  the  Orbital  Transfer  Vehicle  Trade  Space,"  AIAA  Space  2003, 
Long  Beach,  CA,  September  2003. 
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and  "roll-ups"  of  multiple  design  alternatives.  The  left  plot  shows  wet  mass  vs  cost  and  the 
right  plot  shows  delta-V  available  vs  cost  for  a  theoretical  Space  Tug.  The  histograms  at  the  top 
and  bottom  of  each  chart  show  a  binned  view  of  the  data  projected  from  the  x  and  y  axis. 


Figure  13.  Example  of  brushing  and  linking  between  the  coordinated  data  views. 


In  Figure  13,  a  brushing  tool  has  been  used  to  draw  a  lasso  around  design  points  of  interest  in 
the  right  chart  and  the  left  chart  updates  to  reflect  the  new  constrained  design  space.  The 
histograms  also  update  to  provide  the  designer  with  immediate  feedback  on  the  effects  of  the 
constraint.  This  allows  the  designer  to  recognize  more  quickly  whether  restrictions  they  place 
on  a  certain  variable  have  impacts  they  might  not  have  intended  on  other  variables.  The  list  at 
the  bottom  shows  the  remaining  available  designs  and  their  characteristics. 
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Figure  14.  Histogram  filtering. 


The  designer  can  further  restrict  the  design  space  by  clicking  on  the  bins  of  histograms  (Figure 
14).  Note  that  the  bin  second  from  the  left  in  the  left-top  histogram  has  been  clicked  to 
constrain  the  design  space  in  this  example.  This  coupled  with  the  brushing  constraint  has 
restricted  the  design  space  to  only  a  few  designs. 


Figure  15.  Using  hexagonal  binning  to  represent  large  numbers  of  designs. 


An  issue  with  tools  of  this  type  is  scalability  due  to  processing  and  rendering.  For  iEEA  we  will 
want  to  look  at  many  more  design  alternatives  than  were  previously  possible  using  traditional 
EEA  applications.  Figure  15  demonstrates  how  a  design  space  of  1,000,000  alternatives  could 
be  rendered  more  effectively  through  2-D  hexagonal  binning.  Both  charts  plot  the  same 
randomly  generated  design  data.  The  left  chart  shows  the  traditional  scatter  chart  view  of  the 
data  which,  due  to  the  density  of  points,  leaves  many  of  the  points  occluded.  The  chart  on  the 
right  shows  the  binned  version  with  the  density  of  points  in  an  area  encoded  by  color.  The 
right-hand  chart  renders  much  more  quickly  due  to  the  reduced  number  of  polygons  and  allows 
the  designer  to  filter  the  design  and  epoch  space  much  more  quickly. 
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In  the  next  phase  of  the  research,  prototyping  will  continue,  leveraging  additional  techniques 
from  visual  analytics  and  more  explicitly  addressing  hypotheses  1  and  3  in  addition  to  2.  The 
key  benefits  are  expected  to  result  from  having  a  human  in  the  loop  and  many  more  design, 
context  and  needs  alternatives  to  consider.  As  a  consequence,  our  current  working  hypothesis 
is  that  iEEA  enables  improved  decision-making  relative  to  traditional  EEA  because: 

1.  It  applies  advanced  techniques  from  parallel  computing,  OLAP  and  new  visualization 
techniques  to  allow  consideration  of  many  more  alternatives  the  previously  possible. 
These  techniques  also  control  growth  of  data  size  and  improved  processing  of  high¬ 
dimensional  data. 

2.  Data  elicitation  on  possible  epochs  and  design  variable  ranges  of  interest  is  significantly 
improved  through  interactive  techniques  that  allow  a  human-in-the-loop  to  identify 
patterns,  areas  of  interest  or  errors  in  the  space  of  alternatives. 

3.  Improved  intuition  and  buy-in  is  achieved  from  the  fact  that  many  more  alternatives  can 
now  be  considered  and  interactive  approaches  allow  decision-makers  to  more  quickly 
understand  impacts  of  design  decisions  on  performance  across  epochs.  Understanding 
that  certain  design  choices  make  a  design  resilient  to  some  contexts,  but  much  more 
brittle  in  others  is  a  key  insight. 


Next  Steps 

In  Phase  2,  work  will  continue  investigating  framing  and  implementation  challenges  uncovered 
in  Phase  1,  with  additional  prototype  software  tools  to  be  developed.  Depending  on  progress, 
some  iEEA  capability  may  be  incorporated  into  the  IVTea  Suite  MPT  (described  in  later  section 
of  this  report).  One  or  more  case  studies  will  be  started  to  work  through  the  development  of 
visualizations  and  supporting  infrastructure  for  iEEA. 
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Value-Model  Tradeoff 


One  of  the  key  challenges  identified  in  preliminary  research  for  IMCSE  involves  understanding 
the  role  that  model  choice  plays  in  the  generation  and  analysis  of  data  for  decision  making. 
IMCSE  anticipates  making  a  key  contribution  in  terms  of  framing  this  challenge  and  insights 
gained  when  actively  trading  models  as  a  part  of  a  study. 


Background 


Figure  16  depicts  the  general  relationship  between  decision  problems  and  decision  solutions  as 
they  relate  to  data  and  models  for  IMCSE.  In  this  figure,  decision  problems  suggest  a  space  of 
potential  solutions,  which  span  a  design  space.  The  design  space  is  then  sampled  and  evaluated 
through  two  types  of  models:  cost  models  and  performance  models.  Cost  models  seek  to 
predict  the  resources  need  to  develop  and  operate  each  of  the  evaluated  potential  systems. 
Typically  these  estimates  are  in  terms  of  dollars,  and  potentially  time  (i.e.  schedule). 
Performance  models  seek  to  predict  the  operational  behavior  in  context  of  the  evaluated 
potential  systems.  Value  models  seek  to  map  the  resulting  resource  and  performance 
predictions  into  decision-friendly  perceived  benefit  and  cost  metrics.  Value  models  can  be 
simple  (e.g.  just  pass  through  the  cost  and  performance  measures),  or  complex  (e.g.  aggregate 
perceived  benefit  under  uncertainty  of  a  large  number  of  measures).  Each  of  these  models, 
and  the  artificial  data  generated  by  them,  can  be  potentially  altered  by  changes  in  the  epoch 
space  (e.g.  exogenous  context  and  needs  changes).  Updating  occurs  when  users  seek  to  modify 
the  space  definitions,  or  the  models,  in  order  for  them  to  better  address  the  problem  under 
consideration  (or  to  improve  the  trust  or  truthfulness  (i.e.  validity)  of  the  models  and  data). 


Decision 

[Problem] 


Decision 

[Solution] 


During  this  phase  of  the  research,  the  team  has  begun  exploratory  work  defining  model  types 
and  formulation  of  how  model  trading  might  be  implemented.  Leveraging  insights  from  Ricci 
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et  al.  (2014)®^  which  described  the  role  of  interactivity  in  refining  a  user's  captured  value 
model,  we  generalized  the  concept  of  "value  model  trading"  from  tuning  parameters  within  a 
particular  value  model  (i.e.  utility  function  shapes  and  weights  for  a  Multi-Attribute  Utility  value 
model)  to  also  include  trading  of  value  model  formulations  as  well.  There  are  many  possible 
value  models  (e.g.  see  Ross  et  al.  (2010)^°).  For  this  demonstration,  four  alternative  value 
models  were  used:  Multi-Attribute  Utility  (MAU),  Analytic  Hierarchy  Process  (AHP),  Cost-Benefit 
Analysis  (CBA),  and  Measure  of  Effectiveness  (MOE)  (see  Figure  17  below).  Recall  a  value 
models  attempt  to  predict  how  a  particular  decision  maker  might  perceived  net  benefits  and 
costs  for  alternatives  under  consideration.  Different  value  models  treat  the  mapping  of  raw 
data  to  perceived  costs  and  benefits  differently.  For  illustration  purposes,  we  treated  perceived 
costs  as  just  lifecycle  cost  (essentially  as  a  single  dimensional  metric  of  perceived  cost),  while 
we  varied  the  perceived  benefit  model  from  MAU  to  AHP  to  CBA  to  MOE.  The  results  of  this 
variation  were  analyzed  in  terms  of  how  the  set  of  most  perceived  benefit  versus  perceived 
cost  efficiency  changed.  This  was  calculated  as  the  Pareto  efficient  set  for  given  value  models. 
The  sets  were  then  compared  to  see  the  impact  of  value  model  choice  on  proposed  "best" 
alternative  solutions.  This  demonstration  case  utilized  the  IVTea  Suite  software  environment 
being  developed  as  an  IMCSE  supporting  MPT  (described  below). 


Decision 

{Problem) 


Decision 

{Solution) 


Figure  17.  Various  value  models  for  demonstration  case 


Demonstration  of  Value  Model  Trading:  Space  Tug 

For  this  exploratory  case,  the  problem  is  framed  as  the  following: 

A  decision  maker  has  a  budget  for  an  orbital  transfer  vehicle  (a.k.a.  "Space  Tug")  and  thinks  he 
knows  what  he  wants  (in  terms  of  attributes  of  goodness  in  a  system  alternative).  But  he  is 


Ricci,  N.,  Schaffner,  M.A.,  Ross,  A.M.,  Rhodes,  D.H.,  and  Fitzgerald,  M.E.,  "Exploring  Stakeholder  Value  Models 
Via  Interactive  Visualization,"  12th  Conference  on  Systems  Engineering  Research,  Redondo  Beach,  CA,  March  2014. 
™  Ross,  A.M.,  O'Neill,  M.G.,  Hastings,  D.E.,  and  Rhodes,  D.H.,  "Aligning  Perspectives  and  Methods  for  Value-Driven 
Design,"  AIAA  Space  2010,  Anaheim,  CA,  September  2010. 
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aware  that  he  may  not  have  formulated  his  value  model  correctly.  He  wants  to  explore  three 
types  of  uncertainties  in  his  value  model: 

1.  What  value  model  best  represents  his  preferences? 

2.  What  parameters  for  a  given  value  model  best  represent  his  preferences? 

3.  What  if  he  really  doesn't  know  what  his  true  preferences  are  and  wants  instead  a  robust 
solution? 

The  second  question  was  partly  addressed  in  Ricci  et  al.  (2014)^\  and  will  be  done  in  this  study 
in  Phase  2.  The  first  and  third  questions  are  investigated  in  this  exploratory  case  during  Phase  1. 

Approach:  Use  four  different  value  models  to  evaluate  and  represent  benefit  vs.  cost  tradeoffs; 
identify  the  most  value  efficient  alternatives  under  different  value  models;  compare  preferred 
alternatives  across  value  models. 

Models  Used  in  the  Study 

The  design  alternatives  and  performance  and  cost  models  for  Space  Tug  are  described  in 
McManus  and  Schuman  (2003)^^.  The  value  models  that  were  used  in  this  study  are  now 
described: 

Multi-Attribute  Utility  (MAU) 

Multi-Attribute  Utility  value  model  generates  an  aggregate  measure  across  multiple  criteria 
(called  attributes).  Each  of  the  attributes  have  single  attribute  utility  functions  that  map 
attribute  level  to  perceived  benefit  under  uncertainty  of  that  attribute  (typically  quantified  on  a 
zero  to  one  scale).  Each  of  the  single  attribute  utility  functions  are  then  aggregated  via  a  multi¬ 
linear  function  into  a  multi-attribute  utility  score. 

The  equation  for  multi-attribute  utility  is  as  follows: 

^  /<'  =  -!  +  -ki  +  l) 

Here  K  is  the  normalization  constant,  f/(^)  is  the  aggregate  utility  value  across  the  multiple 
single  attributes  X,  and  their  respective  single  attribute  utilities,  Ui{Xi))  kj  is  the  swing  weighting 
factor  for  the  attribute  X,;  and  n  is  the  number  of  attributes.  Figure  18  illustrates  the  three 
single  attribute  utility  functions  for  each  of  the  attributes  (capability,  delta  V,  and  response 
time),  along  with  their  kj  weights  for  the  multi-attribute  utility  function.  In  the  special  case 
where  the  weights  add  to  1,  the  function  becomes  a  linear  weighted  sum,  and  therefore  each 
attribute  contributes  independently  to  the  aggregate  value. 


Ricci,  N.,  Schaffner,  M.A.,  Ross,  A.M.,  Rhodes,  D.H.,  and  Fitzgerald,  M.E.,  "Exploring  Stakeholder  Value  Models 
Via  Interactive  Visualization,"  12th  Conference  on  Systems  Engineering  Research,  Redondo  Beach,  CA,  March  2014. 

McManus,  H.,  and  Schuman,  T.,  "Understanding  the  Orbital  Transfer  Vehicle  Trade  Space,"  AIAA  Space  2003, 
Long  Beach,  CA,  September  2003. 
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Figure  18.  Single  attribute  utility  functions  for  the  MALI  value  model 


Each  of  the  Space  Tug  design  alternatives  were  then  evaluated  in  terms  of  the  MAD  benefit  and 
cost  and  are  plotted  in  Figure  19.  Additionally,  the  Pareto  Efficient  set  of  designs,  which  are  the 
most  benefit-cost  efficient  solutions,  non-dominated  in  this  two  objective  space,  are  indicated 
with  blue  triangles  (flat  side  on  bottom).  Due  to  the  nature  of  multi-attribute  utility,  design 
alternatives  that  do  not  meet  minimum  acceptable  levels  in  any  particular  attribute  are 
deemed  unacceptable  and  are  treated  as  infeasible.  This  results  in  a  smaller  set  of  designs  to 
consider  (here  as  N=83,  out  of  the  total  possible  of  384).  The  designs  in  the  Pareto  Set  did  not 
share  many  common  features,  but  all  had  propulsion  systems  that  were  electric  (type  3)  or 
nuclear  (type  4). 
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Figure  19.  MALI  benefit  versus  Cost  tradespace  with  Pareto  Efficient  designs  indicated 


Analytic  Hierarchy  Process  (AHP) 

Analytic  Hierarchy  Process  value  model  generates  an  aggregate  measure  across  multiple 
criteria.  Each  of  the  criteria  are  evaluated  pair-wise  to  determine  relative  value  contribution. 
The  aggregate  AHP  score  is  determined  using  a  linear-weighted  sum,  where  the  weights  are 
derived  from  the  pairwise  comparisons. 

The  equation  for  AHP  value  is  as  follows: 

AHP{X)  =  Y?=iki  ■  where 

AHPiiXi)  =  ^  if  bigger  is  better  forX,- 

(X'  “X'l 

AHPi^Xi)  =  ^  if  smaller  is  better  for  X/,  and  the  k;  weights  are  determined  from  the 

AHP  matrix. 

vn  %q 

where  Opq  is  the  element  in  row  p,  column  q  in  the  AHP  matrix,  n  is  the  number 
of  criteria  (i.e.  number  of  rows  and  columns  in  the  matrix). 

Figure  20  illustrates  the  pair-wise  comparison  matrix  for  the  three  criteria  (capability,  delta  V, 
and  response  time),  which  resulted  in  weights  of  0.4,  0.4,  and  0.2  respectively. 
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Figure  20.  Matrix  of  weights  for  the  AHP  value  model 


Higher  is  Better 


Lower  is  Better*^ 


Each  of  the  Space  Tug  design  alternatives  were  then  evaluated  in  terms  of  the  AHP  benefit  and 
cost  and  are  plotted  in  Figure  21.  Additionally,  the  Pareto  Efficient  set  of  designs,  which  are  the 
most  benefit-cost  efficient  solutions,  non-dominated  in  this  two  objective  space,  are  indicated 
with  green  triangles  (flat  side  on  right).  Due  to  the  nature  of  AHP  value,  no  design  alternatives 
are  rejected,  so  the  full  tradespace  appears  feasible  (N=384).  The  designs  in  the  Pareto  Set  have 
no  common  patterns  except  they  never  have  electric  propulsion  (type  3). 
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Figure  21.  AHP  benefit  versus  Cost  tradespace  with  Pareto  Efficient  designs  indicated 


Cost-Benefit  Analysis  (CBA) 

Cost-Benefit  Analysis  value  model  converts  multiple  criteria  into  a  common  currency  (typically 
dollars)  in  order  to  simplify  comparisons.  In  order  to  construct  this  model,  one  must  create 
currency  conversion  functions  for  each  of  the  criteria.  For  this  case  demonstration,  each 
conversion  function  has  three  parameters,  which  assumes  a  minimum  acceptable  level  (zero),  a 
marginal  dollar  per  unit  of  the  attribute  (the  conversion  rate),  and  (optionally)  a  diminishing 
returns  rate  (if  the  marginal  rate  decreases  with  an  increase  in  attribute  level).  After  calculating 
each  individual  criterion  as  a  dollar  figure,  the  aggregate  is  a  simple  sum  of  the  three. 

The  equation  for  CBA  value  is  as  follows: 

CBA{x)=Y.UCBMXi), 

CBAi(Xi)  ——(1  —  when  X,  >=  Xi^-m  and 

CBAi(Xi)  =  0,  when  X,  <  Ximin 

Where  m,  is  the  marginal  rate  of  dollars  per  unit  attribute,  r,-  is  the  (optional)  diminishing  return 
rate,  and  Xmin  is  the  minimum  acceptable  level  (or  zero  point)  for  bigger  is  better  functions. 
When  there  is  no  diminishing  returns  rate,  CBA  function  is  simply  a  linear  function  of  m,  X,. 
Figure  22  illustrates  the  three  monetization  functions  for  the  three  criteria  (capability,  delta  V, 
and  response  time). 
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Figure  22.  Attribute  monetization  functions  for  the  CBA  value  model 


Higher  is  Better 


Lower  is  Better 


Each  of  the  Space  Tug  design  alternatives  were  then  evaluated  in  terms  of  the  CBA  benefit  and 
cost  and  are  plotted  in  Figure  23.  Additionally,  the  Pareto  Efficient  set  of  designs,  which  are  the 
most  benefit-cost  efficient  solutions,  non-dominated  in  this  two  objective  space,  are  indicated 
with  red  triangles  (flat  side  on  left).  Due  to  the  nature  of  CBA  value,  no  design  alternatives  are 
rejected,  so  the  full  tradespace  appears  feasible  (N=384).  The  designs  in  the  Pareto  Set  tend  to 
have  small  payloads,  never  have  electric  propulsion  (type  3),  and  otherwise  vary  in  their  design 
variable  levels. 


Figure  23.  CBA  benefit  versus  Cost  tradespace  with  Pareto  Efficient  designs  indicated 
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Measure  of  Effectiveness  (MOE) 


Delta  V  was  used  as  a  single  dimension  Measure  of  Effectiveness  since  it  represents  the 
fundamental  capability  for  transferring  target  vehicles  from  one  orbital  slot  to  another.  Each  of 
the  Space  Tug  design  alternatives  were  evaluated  in  terms  of  the  MOE  benefit  and  cost  and  are 
plotted  in  Figure  24.  Additionally,  the  Pareto  Efficient  set  of  designs,  which  are  the  most 
benefit-cost  efficient  solutions,  non-dominated  in  this  two  objective  space,  are  indicated  with 
cyan  triangles  (flat  side  on  top).  Due  to  the  nature  of  MOE  value,  no  design  alternatives  are 
rejected,  so  the  full  tradespace  appears  feasible  (N=384).  The  designs  in  the  Pareto  Set  tend  to 
have  electric  propulsion  since  this  choice  results  in  the  largest  delta  V  for  a  given  mass 
spacecraft.  All  of  the  designs  also  have  the  minimum  size  payload,  which  again  reduces  the 
overall  dry  mass  of  the  spacecraft,  resulting  in  additional  delta  V  capability  for  the  Space  Tug  to 
impart  on  target  spacecraft. 
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Figure  24.  MOE  (DeltaV)  versus  Cost  tradespace  with  Pareto  Efficient  designs  indicated 


Results 

Now  that  each  of  the  Space  Tug  designs  have  been  evaluated  with  each  of  the  value  models 
and  each  suggests  a  particular  set  of  value  efficient  designs,  the  next  step  is  to  compare  Pareto 
Sets  across  the  four  value  models. 

Comparisons  via  Pareto  Sets 

Figure  25  illustrates  the  symbol  key  for  the  key  comparison  charts  to  follow. 
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MAU-Cost  Pareto  Set  ^ 
AHP-Cost  Pareto  Set  < 
CBA-Cost  Pareto  Set  ► 
MOE-Cost  Pareto  Set  V 


Figure  25.  Key  for  various  value  model  benefit  versus  cost  Pareto  Efficient  sets 


Figure  26.  Comparison  of  four  value  tradespaces 

Figure  26  illustrates  the  four  perceived  benefit  versus  cost  tradespaces  across  the  four  value 
models,  with  all  four  Pareto  sets  indicated.  Upon  inspection,  it  appears  that  no  single  point 
appears  in  all  four  Pareto  sets,  but  there  are  a  few  that  appear  in  three  out  of  four.  The  next 
step  was  more  formal  joint  Pareto  set  analysis  to  determine  the  specifics  of  apparently 
attractive  designs. 

Joint  Pareto  Analysis 

The  Joint  Pareto  analysis  entailed  determining  the  Pareto  Set  for  each  of  the  four  pairs  of 
objectives  (i.e.  benefit  and  cost  functions  for  each  of  the  four  value  models).  The  number  of 
valid  designs,  along  with  each  Pareto  Set  size  is  indicated  in  Figure  27.  It  is  important  to  note 
that  there  are  zero  "joint"  designs.  Here,  "joint"  means  that  the  design  appears  in  all  individual 
Pareto  Sets.  Instead,  there  are  some  "compromise"  designs,  which  are  determined  by 
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calculating  the  Pareto  Set  across  the  union  of  all  objective  functions.  These  represent  efficient 
solutions  that  are  non-dominated  across  the  full  set  of  objectives. 


Objective  Sets: 
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Figure  27.  Joint  Pareto  analysis  set  up  with  four  objective  sets  of  two  objectives  each 


Upon  close  inspection,  we  found  that  there  are  six  designs  that  are  in  three  out  of  four  Pareto 
Sets.  These  are  listed  in  Figure  28,  but  two  of  the  six  are  invalid  for  the  MAU  value  model 
(meaning  they  do  not  provide  minimum  acceptable  benefit  in  one  or  more  attributes). 


ID  Number 

Pareto  Efficient  For 

Invalid  For 

1 

2,  3,4 

1 

11 

2,  3,  4 

1 

63 

1,  2,3 

95 

1,  2,3 

127 

1,  2,  3 

128 

1,  2,  3 

Figure  28.  Promising  designs  that  are  Joint  Pareto  efficient  across  three  out  of  four  value  models 

The  details  of  the  promising  designs  are  described  in  Figure  29.  If  we  do  not  consider  designs  1 
and  11,  which  are  invalid  for  the  MAU  value  model,  we  see  a  few  common  design  choices 
among  the  remainder  of  the  designs.  They  all  use  propulsion  system  type  4,  and  a  large  amount 
of  fuel. 
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Figure  29.  Details  on  the  "promising"  designs 


One  other  technique  we  can  leverage  in  trying  to  find  "robust"  solutions  that  are  insensitive  to 
value  model  choice  is  to  calculate  fuzzy  Pareto  Efficient  sets.  We  varied  the  fuzziness  level  and 
found  that  a  single  design  does  appear  to  be  fully  joint  Pareto  efficient  at  a  fuzzy  level  of  7%. 
This  means  this  design  is  within  7%  of  Pareto  Efficiency  for  all  four  value  models. 
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Figure  30.  Comparison  of  benefit  versus  cost  tradespaces  with  compromise,  promising,  and  fuzzy  joint  designs 
indicated 


Figure  30  illustrates  the  four  tradespaces  again,  this  time  with  the  compromise  (pink 
diamonds),  promising  (yellow  five-pointed  stars),  and  7%  fuzzy  joint  (black  six-pointed  star) 
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designs.  Design  52  is  the  single  7%  fuzzy  joint  Pareto  design  and  represents  the  most  robust 
choice  if  the  decision  maker  is  unsure  of  which  of  the  four  value  models  best  captures  his 
preferences.  Interestingly  this  design  uses  electric  propulsion,  which  was  a  design  choice  absent 
from  the  Pareto  sets  of  AHP  and  CBA  value  models.  Appealingly,  this  design  also  appears  in  the 
low  cost  region  of  all  of  the  tradespaces. 

Next  Steps 

This  exploratory  demonstration  case  for  value  modeling  trading  was  intended  to  help  create 
supporting  infrastructure  and  processes  for  model  trading  capabilities  more  generally.  In  this 
sense,  the  case  was  quite  successful  in  that  it  resulted  in  the  ability  to  use  different  value  model 
formulations  within  the  IVTea  Suite  software.  In  Phase  2,  we  will  continue  the  analysis  of  the 
value  model  trades  in  this  case,  along  with  developing  a  more  complete  framework  and  process 
for  how  to  conduct  value  model  trades  more  generally. 
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Supporting  MPTs 


During  research  activities  within  IMCSE,  a  number  of  opportunities  to  develop  supporting 
methods,  processes,  and  tools  have  arisen.  In  addition  to  the  three  specific  projects  within  the 
three  thrusts  of  the  IMCSE  program,  these  MPTs  will  contribute  to  the  IMCSE  body  of 
knowledge  and  facilitate  knowledge  transfer  to  practice.  During  this  phase,  a  software  suite 
called  IVTea  Suite  was  augmented,  to  facilitate  the  ongoing  IMCSE  research. 


IVTea  Suite 

IVTea  Suite  (Interactive  Value-Driven  and  Tradespace  Exploration  and  Analysis  Suite)  is  a 
software  package  developed  in  MATLAB  at  the  MIT  Systems  Engineering  Advancement 
Research  Initiative  (SEAri).  It  is  intended  to  help  engineering  analysts,  stakeholders,  and 
decision  makers  uncover  insights  about  their  systems  and  support  value  robust  decision  making 
in  the  face  of  large,  uncertain  problems.  The  software  is  a  research  support  tool,  and  not 
intended  for  broad  circulation  as  a  final  product  for  users. 

Motivation 

Development  of  IVTea  began  in  2009  under  the  name  VisLab  (Visualization  Laboratory).  The 
original  vision  for  VisLab  was  to  create  a  platform  leveraging  the  research  library  of  SEAri  and 
allow  for  the  effective  reuse  of  data  and  advanced  tradespace  visualizations  without  the  need 
to  'reinvent  the  wheel'  for  every  project.  Interactive  software  providing  real-time  feedback 
could  reduce  the  delay  between  imagining  questions  and  finding  answers,  thus  accelerating  the 
development  of  insight  into  the  systems  of  interest.  Additionally,  the  promise  of  a  highly 
modular  code  base  could  enable  graduate  students  to  contribute  individual  'widgets',  thus 
rapidly  and  easily  expanding  the  software's  capabilities  over  time  as  new  techniques  were 
created  at  MIT  SEAri. 

During  the  development  of  VisLab  1.0,  the  key  vision  captured  by  the  software  was  one  of 
supporting  epoch-centric  analysis:  the  visualization  and  analysis  of  the  different  tradespaces 
created  by  varying  the  context  and  preferences  under  which  the  system  operates.  As  SEAri 
research  began  to  expand  more  heavily  into  multi-epoch  and  era  analysis  (across  all  uncertainty 
and  across  time-dependent  sequences  of  uncertainty,  respectively),  it  became  apparent  that 
VisLab  would  require  considerable  architecture  upgrades  in  order  to  handle  these  advanced 
analysis  types.  VisLab  2.0  and  subsequently  IVTea  1.0  have  gradually  improved  the  architecture 
and  user  experience  of  the  software,  now  supporting  all  of  these  analyses  and  providing  a 
comprehensive  set  of  perspectives  from  which  to  view  the  design  problem. 

Relationship  between  IVTea  and  IMCSE 

IMCSE  is  intending  to  dramatically  enhance  the  knowledge  base  and  capabilities  of  systems 
engineers  who  interact  with  models.  Models  are  a  critical  part  of  systems  engineering,  usually 
providing  all  or  nearly-all  of  the  data  available  to  assist  early  concept  design  and  decision 
making.  The  models  are  necessary  because  systems  engineers  typically  work  on  problems  for 
which  it  is  infeasible  to  construct  prototypes  and  simply  test  or  benchmark  data  relevant  to  the 
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decisions  they  need  to  make,  and  thus  must  either  collect  natural  data  to  form  empirical 
models  or  generate  artificial  data  from  a  theoretical  model.  IMCSE  acknowledges  this  core 
dependence  and  seeks  to  empower  systems  engineers  to  better  understand  how  they  interact 
with  models  and  leverage  their  strengths  while  respecting  their  weaknesses. 

IVTea  supports  the  goals  of  IMCSE  by  allowing  for  direct  feedback  between  engineers  and 
models.  IVTea  can  run  on  fractional  datasets  and  provide  updates  on  the  progress  of  models 
running  in  the  background,  in  order  to  allow  analysis  to  occur  before  the  completion  of  all 
simulations  which  may  sometimes  take  days,  weeks,  or  more.  IVTea  also  fully  supports  the 
real-time  creation  and  modification  of  value  models  applied  to  the  system.  Multi-Attribute 
Utility  Theory,  the  Analytic  Hierarchy  Process,  Cost-Benefit  Analysis,  and  simple  Measure  of 
Effectiveness  models  are  all  present  in  the  IVTea  architecture  and  can  be  compared  side-by- 
side,  modified,  or  swapped  on  the  fly  in  order  to  understand  the  impact  that  different  models 
of  value  (and  their  associated  parameters)  have  on  the  tradespace.  IVTea  also  features  partial 
support,  to  be  developed  further  in  the  future,  for  the  tradeoffs  between  performance  models 
and  simulations,  enabling  the  visualization  of  tradespaces  at  varying  levels  of  fidelity  or  with 
different  assumptions. 

Capabilities  of  IVTea 

IVTea  provides  visualization  and  analysis  capabilities  for  the  interrogation  of  performance  and 
value  models.  The  requirements  to  use  IVTea  for  a  case  study  are  (1)  a  performance  model  or 
models  (either  natural  or  artificial)  that  can  evaluate  potential  design  alternatives  and  (2)  a 
properly  structured  MySQL  database  populated  by  the  input/output  data  of  that  model.  The 
IVTea  database  structure  has  only  seven  mandatory  tables  and  is  easily  formatted. 

IVTea  reads  data  from  a  specified  database  and  displays  it  through  the  use  of  a  variety  of 
'widgets'.  Widgets  are  tools  or  visualizations  specifically  designed  to  perform  one  task,  with  the 
full  set  of  widgets  providing  the  complete  functionality  of  IVTea  when  used  together.  Widgets 
are  linked  not  only  by  the  underlying  database  data  but  by  locally-stored  session  data  as  well. 
This  allows  information  such  as  favorite  designs  or  contexts  of  interests  to  be  propagated 
between  analysis  tools  quickly  and  effectively.  Session  data  can  also  be  saved  in  a  file  and 
loaded  at  a  later  time  to  resume  an  in-progress  case  study  without  losing  valuable  information. 

The  main  window  of  IVTea  is  called  the  Dashboard.  The  Dashboard  is  the  interface  for  opening 
new  widgets  and  managing  session-level  commands  such  as  saving  data  or  loading  a  new 
project.  Figure  31  shows  the  Dashboard  in  its  standard  view.  Each  square  button  opens  a 
corresponding  widget,  and  the  list  on  the  right  shows  all  currently  open  widgets,  allowing  for 
easy  managements  of  active  windows.  The  widget  panel  is  tabbed,  separating  widgets  into 
categories  based  on  the  type  of  analysis  they  most  often  support:  design-centric,  epoch-centric, 
or  era-centric,  in  addition  to  data  management  widgets  and  prototype  widgets.  Also  note  the 
top  bar  which  has  a  few  key  feature  consistent  across  all  widgets.  Options  contains  a 
'screenshot  window'  function.  View  allows  for  customization  of  widgets  based  on  user  type, 
and  Help  opens  an  interactive  HTML  help  directory  for  IVTea,  including  screenshots  and  usage 
instructions  for  all  of  the  widgets. 
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Figure  31.  The  IVTea  Dashboard 

The  following  is  a  list  of  the  current  widgets  in  IVTea  with  a  brief  description  of  their 
capabilities: 

Epoch-Centric  Analysis 

Epoch-centric  analysis  focuses  on  one  (or  a  few)  epoch(s)  at  a  time,  for  example  looking  at  all 
designs'  performances  within  a  given  epoch.  The  widgets  in  this  category  tend  to  focus  on 
picking  one  or  a  few  epochs,  and  then  representing  and  helping  users  to  analyze  within  that  set. 

•  Epoch  Filter  -  Find  the  set  of  epochs  that  obey  a  user-specified  group  of  logical 
statements  about  their  context. 

•  Epoch  Knobs  -  See  the  defining  context  and  preference  variables  of  an  epoch,  and  easily 
modify  them  to  find  similar  epochs. 

•  Tradespace  Viewer  -  The  standard  tradespace  view  shows  all  valid  design  alternatives 
for  a  specified  epoch  as  points  on  a  graph.  The  x,  y,  z,  color,  and  size  axes  are  all 
customizable  to  display  different  variables  in  the  database.  Hotkeys  are  available  to 
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snap  axes  to  the  benefit/cost  view  of  different  stakeholders.  The  scatterplot  has  pan, 
zoom,  rotate,  and  group  brush  tools,  as  well  as  the  ability  to  right-click  design  points  to 
bring  up  a  context  menu  with  information  about  that  design  and/or  save  it  as  a  favorite. 
Figure  32  gives  an  example  of  this  widget. 


Figure  32.  Atradespace  with  benefit/cost  axes,  colored  by  design  variable,  with  two  highlighted  favorite  designs 

•  Context  Space  Viewer  -  Create  a  grid  of  scatterplots  and  histograms  showing  the 
enumeration  scheme  and  completeness  of  the  context  space. 

•  Carpet  Plot  -  Create  a  grid  of  scatterplots  showing  the  effect  of  each  design  variable  (on 
the  x-axis)  against  each  performance  attribute  (on  the  y-axis).  This  view  is  useful  for 
verifying  intended  variable  interactions  in  the  models  or  uncovering  unexpected 
interaction. 

•  Preference  Explorer  -  View  the  specified  preferences  of  each  active  decision  maker  in  a 
specified  epoch.  This  includes  capability  for  all  of  the  different  value  models  included  in 
IVTea  (MALI,  AHP,  CBA,  and  MOE),  each  with  an  accompanying  interface.  The 
preferences  can  be  modified  and  stored  locally,  shared  among  the  other  widgets.  Figure 
33  gives  an  example  of  this  widget. 
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Figure  33.  The  preference  explorer  viewing  two  attributes  of  a  Multi-Attribute  Utility  function 

•  Pareto  -  Specify  objective  sets  and  find  the  designs  which  are  Pareto  efficient  across 
them  for  a  given  epoch.  Objective  sets  can  include  any  number  of  objectives  greater 
than  one.  Pareto  sets  can  be  modified  with  allowances  for  fuzziness,  can  be  compared 
to  find  joint-  and  compromise-efficient  designs,  and  can  be  easily  saved  as  favorites. 

•  DV Streaks  -  Shows  a  standard  tradespace  view  but  adds  the  ability  to  draw  'streaks' 
between  designs  that  are  the  same  except  for  a  single,  specified  design  variable.  Streaks 
can  be  applied  to  manually  or  to  favorites,  and  are  customizable.  This  view  shows  the 
sensitivity  of  designs  to  perturbations  in  their  variables. 
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Design-Centric  Analysis 

Design-centric  analysis  focuses  on  one  (or  a  few)  clesign(s)  at  a  time  for  analysis,  for  example 
looking  at  how  a  design  performs  across  all  epochs.  The  widgets  in  this  category  tend  to  focus 
on  picking  one  or  a  few  designs,  and  then  representing  and  helping  users  to  analyze  that  set. 

•  Design  Filter  -  Find  the  set  of  designs  that  obey  a  user-specified  group  of  logical 
statements  about  their  design  variables.  Figure  34  gives  an  example  of  this  widget. 


Figure  34.  Two  filters  find  ten  matching  designs  in  the  Design  Filter  tool 

•  Design  Knobs  -  See  the  defining  variables  of  a  design,  and  easily  modify  them  to  find 
similar  designs. 

•  Design  Tradespace  Viewer  -  A  variation  on  the  standard  tradespace  that  shows  each 
epoch  as  a  point  on  the  graph  for  a  single  design  (rather  than  each  design  as  a  point  for 
a  single  epoch).  Has  the  same  features  as  the  Tradespace  Viewer,  but  is  used  to  identify 
the  effects  of  changing  context  on  the  performance  of  one  design. 

•  Design  Space  Viewer  -  Create  a  grid  of  scatterplots  and  histograms  showing  the 
enumeration  scheme  and  completeness  of  the  design  space. 
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•  Comparison  Tool  -  Place  designs  of  interest  into  a  table  allowing  side-by-side 

comparison  of  their  variables  and  performance  attributes.  A  baseline  design  can  be  set, 
coloring  the  other  table  entries  based  on  their  higher/lower  relationship  to  the  baseline. 
Figure  35  gives  an  example  of  this  widget. 


y  Comparison  Tool  i  S  1,  S3  i 

File  Options  Help 


Figure  35.  A  comparison  table  comparing  a  design  to  a  baseline 


•  Fuzzy  Pareto  Number  -  Show  a  histogram  of  the  specified  design's  Fuzzy  Pareto  Number 
(a  measure  of  cost-benefit  efficiency)  across  all  epochs.  This  view  gives  an  overview  of 
the  design's  performance  across  the  complete  uncertainty  space. 

•  Filtered  Outdegree  -  Plot  a  tradespace  and  color  it  with  live  calculation  of  Filtered 
Outdegree  for  specified  designs.  This  illustrates  differences  in  available  change  options 
for  each  design. 

Era-Centric  Analysis 

Era-centric  analysis  focuses  on  one  (or  a  few)  era(s)  at  a  time  analysis,  for  example  looking  at  all 
designs'  performances  across  a  given  era.  The  widgets  in  this  category  tend  to  focus  on  picking 
one  or  a  few  eras,  and  then  representing  and  helping  users  to  analyze  across  set  of  eras. 


•  Era  Constructor  -  Build  an  era  (or  eras)  as  a  sequence  of  epochs  for  use  in  other  widgets. 
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•  Era  Viewer  -  Plot  the  tradespaces  of  each  epoch  in  an  era  side-by-side  on  consistent 
axes. 

•  Morph  -  Animate  the  trajectories  of  each  design  across  the  epochs  in  an  era.  Allows 
playback,  looping,  and  frame-by-frame  stepping  in  addition  to  the  standard  tradespace 
visualization  options.  The  animation  is  particularly  effective  for  helping  users  track  both 
individual  designs  and  overarching  trends. 

Management 

Management  widgets  focus  on  organizing  data  across  the  different  categories  of  analyses. 

•  Favorites  Manager  -  This  widget  keeps  track  of  all  designs  and  epochs  locally  saved  as 
favorites,  and  allows  for  manual  entry  of  new  favorites.  Favorites  can  also  be  saved  as 
batches.  Favorites  have  plotting  options  that  enable  other  widgets  to  display  them  with 
a  consistent,  customizable  marker  (size,  shape,  and  color). 

•  Notes  -  A  text  entry  field  for  keeping  track  of  notes  during  the  session.  Notes  are  saved 
for  the  current  session  even  if  the  widget  is  closed,  and  can  be  permanently  saved  as  a 
text  file. 

•  Summary  Dash  -  Presents  an  overview  of  the  status  of  the  currently  connected 
database.  Includes  totals  for  evaluated  designs,  epochs,  preference  sets,  and  others, 
and  can  be  clicked  to  display  more  detail  about  individual  database  tables.  A  diagram 
view  also  offers  a  visual  representation  of  the  relationships  between  the  tables. 

•  Responsive  System  Comparison  -  A  workflow  outline  of  SEAri's  RSC  method,  which  can 
provide  guidance  for  new  users  on  the  use  of  tradespace  exploration  and  the  relevant 
widgets  for  each  step.  Figure  36  gives  an  example  of  this  widget. 
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D  Responsive  System  Comparison  (  no  ^  I 

p5.  Single  Epoch  Analyses - 

This  process  includes  the  analysis  of  MALI  and  MAE  of  alternatives  within  particular  epochs,  including  designs 
graphically  compared  on  an  MAU  versus  MAE  scatterplot  for  any  given  epoch  (time  period  of  fixed  operating  context 
and  stakeholder  needs).  Within-epoch  metrics,  such  as  yield,  give  an  indication  of  the  difficulty  of  a  particular  context 
and  needs  set  for  considered  designs. 


Figure  36.  An  example  guide  slide  to  step  5  of  RSC 


•  DM  Creator  -  Allows  the  insertion  of  new  decision  makers  to  the  active  database  (if 
appropriate  permission  is  available).  New  DMs  are  assigned  to  new  epochs  that 
replicate  existing  epochs,  but  with  new  preferences. 

•  Preference  Creator  -  Allows  the  insertion  of  new  preference  sets  to  the  active  database 
(if  appropriate  permission  is  available).  This  supports  all  four  value  models  and  allows 
full  customization  of  their  parameters. 

Next  Steps 

Going  forward,  IVTea  Suite  will  continue  to  undergo  refinement  of  user  interface,  data 
handling,  as  well  as  development  of  additional  widgets  that  support  ongoing  research. 
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Applications 


The  applications  thrust  in  this  phase  includes  the  Interactive  Schedule  Reduction  Model  project. 


Interactive  Schedule  Reduction  Model 

Leveraging  prior  work  from  DARPA  META,  the  Schedule  Reduction  Model  will  be  extended  with 
interactivity  as  a  central  aspect,  promoting  sensitivity  analyses  and  benchmarking  to  be  the 
central  use  case. 


Introduction 

Model-based  systems  engineering  planning  environments  with  interactive  capability  hold 
promise  for  accelerating  the  planning  process,  and  doing  better  planning.  Sharon,  de  Week 
and  Dori  (2009)^^  describes  a  model-based  approach  and  the  benefits  to  be  gained  through 
"what-if"  interactive  planning  models.  A  systems  dynamics  model  prototype,  developed  by  de 
Week  (Murray,  et  al.,  2011)^^  in  prior  DARPA  work  explored  the  potential  for  an  interactive 
project  planning  environment.  An  open  area  of  research  is  to  build  on  the  prototype  with 
exploratory  extensions  for  planning. 

Large  engineering  projects  face  continued  risk  of  significant  cost  and  schedule  overruns  despite 
advances  in  technology  and  management  processes.  Industries  involving  aerospace  and 
defense  systems  are  particularly  afflicted.  A  recent  GAO  report  highlights  74  instances  of  cost 
breaches  in  47  of  134  major  defense  acquisition  programs  since  1997.^^  The  largest  factors 
responsible  for  unit  cost  growth  include  engineering  and  design  issues,  schedule  issues,  and 
quantity  changes.  Nearly  40  percent  of  cost  breaches  occurred  after  finalizing  production 
decisions,  further  constraining  options  for  project  restructuring.  Among  other 
recommendations,  a  GAO  testimony  calls  for  early  and  continued  systems  engineering  analysis 
to  identify  and  intervene  before  significant  overruns  occur.^®  Increased  effort  to  consider  design 
alternatives  and  evaluate  achievability  of  objectives  during  early  design  reviews  would  ensure 
the  project  meets  requirements  with  available  resources. 

Proposed  methods  to  reduce  the  risk  of  cost  and  schedule  overruns  may  alter  established 
systems  engineering  and  project  management  processes.  The  META  II  Complex  Systems  Design 
and  Analysis  (CODA)  project  investigated  new  design  techniques  such  as  deliberate  use  of 
layers  of  abstraction,  development  and  use  of  a  component  model  library  (C2M2L),  and  virtual 
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verification  and  validation  processes/^  Due  to  the  long  durations  and  high  cost  of  target 
projects,  it  is  impossible  to  empirically  evaluate  the  effectiveness  of  proposed  changes.  Instead, 
models  of  the  project  development  cycle  assess  alternative  management  schemes.  Previous 
work  developed  the  Design  Flow  Model  to  illustrate  potential  benefits  of  a  META-enabled 
design  process. 

This  project  develops  the  Interactive  Schedule  Reduction  Model  (ISRM)  to  advance  knowledge 
and  experience  by  exploring  alternative  development  processes  and  resource  allocations. 
Additions  to  the  existing  Design  Flow  Model  include  rapid  sensitivity  analysis  of  factors  to 
determine  potential  impact  on  program  schedule  and  development  of  a  customizable  and 
extensible  browser-based  interactive  user  interface.  This  report  discusses  initial  progress 
towards  developing  the  ISRM,  and  reviews  background  literature  and  theoretical  motivation. 
We  introduce  the  approach  for  developing  the  ISRM  as  a  browser-based  application,  and 
describe  initial  progress  to  develop  a  JavaScript  port  of  an  existing  Design  Flow  model.  We 
outline  the  future  work  to  complete  project  objectives. 

Background  and  Motivation 
Project  Complexity 

Aerospace  and  defense  engineering  projects  are  particularly  afflicted  by  cost  and  schedule 
overruns.  This  pattern  has  been  popularized  by  Augustine's  Law  16  which  observes  that  aircraft 
unit  costs  increase  exponentially  while  budgets  increase  linearly,  leading  to  the  seemingly- 
absurd  case  where  the  entire  defense  budget  would  afford  just  one  aircraft  by  2054.^^  A  study 
of  fixed-wing  aircraft  estimates  economy-driven  factors  contribute  only  about  a  third  of  cost 
growth. The  remaining  two-thirds  are  attributed  to  customer-driven  factors  with  major 
contributions  from  complexity  of  performance  characteristics  and  airframe  material. 

There  are  many  descriptions  and  definitions  of  complexity  in  literature;  however,  a  unifying 
perspective  for  system  design  relates  it  to  uncertainty  in  meeting  functional  requirements 
within  cost  and  schedule  constraints  due  to  increases  in  required  information.^®  Sources  of 
complexity  include  structural  (components  and  interrelationships),  behavioral  (functional 
response  to  inputs),  contextual  (outside  circumstances),  temporal  (time  dynamics),  and 
perceptual  (stakeholder  preferences)  aspects.^®  Most  efforts  to  quantify  complexity  focus  on 
structural  aspects.  For  example,  entropy-based  methods  define  a  metric  as  a  function  of  system 
components,  their  interconnections,  and  overall  architecture.^^  In  particular  applications. 
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systems  with  higher  complexity  measures  can  be  shown  to  provide  higher  levels  of 
performance  than  simpler  systems  if  they  are  optimally  managed. 

The  downsides  of  complexity  arise  from  limitations  in  individual  and  social  cognition.  To 
emphasize  this  distinction,  consider  complexity  to  have  descriptive  and  perceived  factors. 
Descriptive  complexity  is  the  objective  system  property  related  to  required  information 
described  above.  Perceived  complexity  is  the  subjective  property  related  to  uncertainty  in 
meeting  requirements  within  constraints  due  to  incomplete  knowledge  of  required  information 
by  an  observer.  This  work  hypothesizes  that  perceived  and  descriptive  complexity  are 
correlated  and  constitute  a  tradeoff  between  design-efficiency  and  design-robustness  also 
observed  in  broader  systems  architecting.^^  Here,  design-efficiency  describes  the  performance 
level  of  functional  requirements  (which  may  increase  with  descriptive  complexity)  and  design- 
robustness  describes  the  certainty  in  achieving  those  functional  requirements  within  cost  and 
schedule  constraints  (which  decreases  with  perceived  complexity). 

Design  studies  consistently  show  a  super-linear  relationship  between  descriptive  complexity 
measures  and  time  to  complete  a  design  with  fixed  requirements. Although  perceived 
complexity  cannot  be  observed  as  a  hidden  intermediate  variable,  this  work  hypothesizes  it  to 
be  the  underlying  mechanism  for  cost  and  schedule  overruns.  Consider  the  illustrative  example 
in  Figure  37.  The  first  plot  shows  a  new  project  with  an  increase  in  descriptive  complexity  to 
meet  higher  performance  requirements  compared  to  past  projects.  The  second  plot  assumes 
perceived  complexity  to  be  related  to  descriptive  complexity  by  a  monotonically-increasing 
function  dependent  on  the  particular  system  and  its  observers.  Differences  in  function  slope 
and  shape,  for  example,  distinguish  between  VLSI  and  mechanical  design. Finally,  the  third 
plot  shows  project  cost  and  schedule  to  be  a  super-linear  function  of  perceived  complexity 
similar  to  findings  of  design  studies. 
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Figure  37.  New  projects  with  high  descriptive  and  perceived  complexity  can  lead  to  large  overruns  on  cost  and 
schedule. 


There  are  three  potential  sources  of  cost  or  schedule  estimation  errors  in  this  model:  errors  in 
level  of  descriptive  complexity,  errors  in  relating  perceived  and  descriptive  complexity,  and 
errors  in  relating  perceived  complexity  and  cost  and  schedule.  However,  the  third  factor  is  the 
most  likely  to  occur  and  has  the  largest  impact  on  large  engineering  projects  for  two  reasons. 
First,  humans  have  difficulty  in  estimating  geometric  or  exponential  growth,  instead  using  linear 
extrapolations  in  intuitive  assessment  which  lead  to  gross  under-estimations  as  shown  by  the 
overly-optimistic  estimate  in  Figure  37.^^  Second,  the  super-linear  growth  of  cost  and  schedule 
magnifies  estimation  errors  for  projects  most  susceptible  to  cost  and  schedule  overruns. 


Design  Methods  and  Tools 


There  are  two  general  approaches  to  address  cost  growth  in  engineering  projects:  decrease 
descriptive  complexity  at  the  cost  of  lower  performance,  or  improve  the  ability  to  perceive  it. 
This  work  takes  the  second  approach  by  leveraging  the  subjective  nature  of  perceived 
complexity.  We  hypothesize  new  methods  and  tools  may  reduce  perceived  complexity  and  help 


designers  acquire  the  required  knowledge  for  descriptively-complex  systems.  Figure  38  shows 
this  effect  by  reducing  perceived  complexity  for  a  given  descriptive  complexity.  Due  to  the 
super-linear  effect  on  cost  and  schedule,  innovations  of  new  design  tools  can  produce 
significant  cost  and  schedule  savings. 
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Figure  38.  New  design  methods  and  tools  may  reduce  project  cost  and  schedule  by  lowering  perceived 
complexity. 

Model-based  systems  engineering  (MBSE)  applies  methods  and  tools  throughout  the  entire 
product  lifecycle  to  replace  labor-intensive,  error-prone,  and  cumbersome  document-based 
processes  with  model-based  methods  more  suitable  to  support  human  cognition  and 
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collaboration.  In  addition  to  basic  efficiency  gains,  MBSE  may  reduce  perceived  complexity  of 
descriptively-complex  systems  and  achieve  greater  cost  and  schedule  reduction  on  complex 
engineering  projects. 

Design  Flow  Model 

The  DARPA  META  II  program  developed  a  method  for  complex  systems  design  and  analysis 
(CODA)  to  implement  several  design  innovations  related  to  MBSE.®^  First,  multiple  layers  of 
abstraction  allow  a  project  to  be  quickly  developed  at  a  coarse  level  and  refined  during  detailed 
design.  Second,  designers  develop  and  maintain  a  trusted  component  model  library  to  limit 
costly  model-building  and  validation  exercises.  Finally,  re-design  cycles  take  place  in  virtual 
environments,  allowing  designers  to  rapidly  evaluate  concepts. 

Past  work  developed  the  Design  Flow  Model  to  evaluate  the  feasibility  of  a  five-fold  speedup  in 
system  development  under  the  META  approach. It  uses  the  System  Dynamics  (SD)  formalism 
to  functionally  specify  stocks  (accumulations)  and  flows  (rates  of  change)  associated  with  a 
product  development  cycle.  Important  stocks  include  requirements  elicited,  architectures 
explored  and  retained,  specifications  generated,  tests  performed,  changes  pending, 
requirements  validated,  and  cost  incurred.  SD  uses  numerical  techniques  to  integrate  stocks  as 
a  system  of  differential  equations  in  a  time-stepped  simulation. 

Results  of  simulated  projects  show  an  idealistic  project  requires  42.25  months  and  $27. 9M  of 
non-recurring  engineering  (NRE)  cost  to  complete.  When  considering  rework  due  to  change 
generation,  however,  a  realistic  project  requires  70  months  and  $51. 9M  in  NRE  costs.  A 
demonstrative  META-enabled  project  with  partial  model  library  completion  requires  only  15.75 
months  and  $31. 5M  in  NRE  costs  -  a  speedup  factor  of  4.4.  Most  performance  gains  are  due  to 
early  design  work  at  higher  levels  of  abstraction  which  catches  problems  earlier  in  the 
development  cycle. 

While  initial  results  are  promising,  further  work  must  determine  the  model's  applicability  to 
other  engineering  projects  and  evaluate  the  sensitivity  of  results  to  input  parameters. 
Furthermore,  as  Sterman  writes,  "effective  learning  from  models  occurs  best,  and  perhaps  only, 
when  decision-makers  participate  actively  in  the  development  of  the  model."®^  Interactive 
"what-if"  planning  models  have  been  shown  to  provide  benefits  in  similar  project  management 
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contexts  and  may  be  effective  to  allow  practitioners  to  understand  and  evaluate  benefits  of 
applied  MBSE  efforts  such  as  META.^^ 

The  Interactive  Schedule  Reduction  Model  (ISRM)  extends  the  Design  Flow  Model  with  two  new 
objectives.  First,  ISRM  develops  new  methods  to  rapidly  generate,  access,  and  interpret  large 
quantities  of  data  generated  from  model  executions,  such  as  those  required  in  sensitivity 
analyses.  Second,  the  existing  SD  model  will  be  transformed  into  a  browser-based  tool  to 
facilitate  interaction  and  extension.  While  its  current  implementation  in  Vensim  is  an  industry- 
standard  SD  modeling  tool,  it  lacks  key  features  for  application  customization  and  methods  to 
generate  and  visualize  large  data  sets  across  many  model  executions.  Furthermore,  as  a 
commercially-licensed  product,  the  current  model  itself  cannot  easily  be  modified  or  broadly 
distributed.  Browser-based  technologies  now  represent  some  of  the  most  innovative 
interactive  tools  and  extensible  models  using  a  commonly-available  platform. 

Approach 

This  project  develops  ISRM  as  an  interactive  browser-based  modeling  and  model  exploration 
tool  in  six  key  steps  completed  under  Phases  1  and  2: 


Phase  1 

Phase  2 

1.  Develop  a 
model  API 

2.  Port 
existing 
model  to 
JavaScript 

3.  Develop  a 
model  Ul 

4.  Develop  a 
multi-model 

API 

5.  Develop 
the  multi¬ 
model 

backend 

6.  Develop  a 
multi-model 

Ul 

Phase  1  Approach 

Phase  1  develops  a  browser-based  ISRM  tool  to  replicate  previous  results  of  the  Design  Flow 
Model  implemented  in  Vensim.  The  resulting  application  allows  users  to  redefine  input 
parameters,  run  a  simulation  execution,  and  view  or  export  numerical  results. 

Step  1  develops  an  application  programming  interface  (API)  for  JavaScript-based  simulation 
models.  Unlike  other  programming  languages  such  as  Java,  Python,  and  MATLAB,  JavaScript  is 
not  frequently  used  for  mathematical  computing  and  may  lack  core  functionality.  Developing  a 
simple  API  sets  the  groundwork  for  future  model  development.  The  API  shall  use  existing 
libraries  when  possible  and  may  identify  new  components  specific  to  modeling  and  simulation 
to  be  developed. 

Step  2  ports  the  existing  Design  Flow  Model  from  Vensim  to  JavaScript  using  the  API  developed 
in  step  1.  The  revised  model  shall  provide  easy  modification  of  input  parameters, 
customization,  and  integration  with  user  interfaces  in  step  3.  Once  complete,  outputs  from  the 


Sharon,  A.,  0.  L.  de  Week,  and  D.  Dori,  "Is  There  a  Complete  Project  Plan?  A  Model-based  Project  Planning 
Approach,"  in  Nineteenth  Annual  International  Symposium  of  the  International  Council  on  Systems  Engineering 
(INCOSE),  Singapore,  2009. 
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ported  model  shall  be  cross-validated  against  existing  model  outputs  under  a  variety  of  inputs 
to  verify  correctness. 

Step  3  develops  a  user  interface  (Ul)  to  allow  interactive  model  exploration  in  a  browser 
environment.  This  step  gives  a  user  control  over  model  execution  and  provides  graphical 
interfaces  for  visualizing  model  structure  and  data  outputs.  In  addition  to  visual  displays,  it  shall 
provide  data  export  for  further  post-processing. 

Phase  2  Approach 

Phase  2  investigates  rapid  sensitivity  analysis  of  various  factors  to  determine  their  potential 
impact  on  project  schedule.  This  phase  develops  an  application  which  allows  users  to  specify 
ranges  of  input  parameters  to  be  considered.  The  application  processes  necessary  model 
executions  to  generate  and  store  required  outputs.  Once  complete,  the  user  visualizes  and 
exports  results  of  the  requested  analysis.  Phase  2  requires  a  different  architecture  from  Phase  1 
to  avoid  long  delays  between  requests  and  results  which  may  include  hundreds  or  thousands  of 
model  executions. 

Step  4  develops  a  multi-model  API  to  aggregate  and  interpret  results  across  many  model 
executions.  It  defines  how  to  configure,  execute,  store,  and  access  results  from  model 
executions.  It  may  rely  on  service-oriented  architectures  to  make  high-performance  model 
execution  or  access  to  a  large  database  of  past  execution  results  accessible  to  clients. 

Step  5  develops  the  backend  components  to  interact  with  the  API  in  step  4.  It  may  include  a 
server-side  model  for  rapid  execution  and  databases  for  storing  and  caching  results.  For 
sensitive  projects,  server  functionality  can  be  included  on  a  protected  local  network  or  as  a 
separate  service  on  a  client  machine.  New  metrics  may  be  developed  to  aggregate  time- 
stepped  result  outputs  to  key  figures  of  interest. 

Finally,  step  6  develops  a  new  Ul  to  allow  uses  to  interact  with  multi-model  data  in  a  client-side 
browser.  New  visualization  and  data  visualization  techniques  shall  be  created  to  show  and 
interpret  large  quantities  of  information  and  support  model  execution  under  conditions  of 
interest. 


Initial  Progress 

This  section  documents  Phase  1  progress  towards  the  ISRM  completing  steps  1-3  above. 

JavaScript  Model  API 

There  are  a  few  existing  modeling  tools  available  to  web  platforms  such  as  JavaScript.  Forio 
Simulate  is  a  commercial  web-based  service  addressing  many  of  the  same  goals  of  this  project; 
however  it  is  closed-source  and  proprietary.®^  Insight  Maker  is  a  similar  open  web-based 
modeling  tool.®^  Rather  than  providing  a  general-purpose  library,  however.  Insight  Maker  is  an 


Forio  Simulate,  http://forio.com/simulate,  accessed  22-Sept  2014. 
Insight  Maker,  http://insightmaker.com,  accessed  22-Sept  2014. 
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integrated  graphical  tool  which  limits  model  extension  and  use  for  other  purposes.  SIM.JS  is  a 
JavaScript  discrete  event  simulation  library  with  support  for  important  features  such  as  random 
number  generation  but  does  not  support  the  SD  formalism.®^  Other  mathematical  computing 
libraries  such  as  Numeric  Javascript  and  Sylvester  implement  features  such  as  vectors  and 
matrixes,  but  do  not  explicitly  provide  numerical  integrators  required  for  the  SD  formalism. 

Based  on  limitations  of  existing  JavaScript  implementation,  the  model  API  develops  a  new 
JavaScript  library  (tentatively  JSIM)  for  SD  models.  It  uses  the  Backbone.js  library  for  object- 
oriented  class  structuring  and  improved  collection  support. Figure  39  shows  an  object  class 
diagram  for  the  API  to  port  models  in  the  SD  formalism.  All  simulation  components  descend 
from  a  common  Entity  class  which  establishes  required  attributes  (id  and  name)  and 
methods  to  initialize  (init),  and  advance  time  (tick/tock).  The  tick  method  pre¬ 
computes  state  changes  to  avoid  order  dependence  which  are  committed  in  the  took  method. 


Figure  39.  Object  class  diagram  for  JSIM  library  API  supporting  the  System  Dynamics  formalism. 

Several  Entity  subclasses  define  components  within  the  SD  formalism.  The  Timer  class 
maintains  the  current  simulation  time.  The  Parameter  class  defines  components  with  a 
constant  value.  The  Flow  class  defines  components  with  value  dependent  on  other 
components,  functionally  defined  by  overriding  the  getValue  method.  Finally,  the  Stock 
class  defines  components  with  a  state  variable  numerically  integrated  during  a  simulation  with 
derivative  specified  by  overriding  the  getDerivative  method.  The  default  integration 
technique  is  explicit  Euler;  however  the  integrate  method  can  be  overridden  to  implement 
alternative  algorithms.  Two  additional  Stock  subclasses  define  specialized  functions  used 
within  SD.  The  Delayl  class  defines  a  first-order  exponential  delay  of  an  input  signal  specified 


SIM.JS,  http://simis.com,  accessed  22-Sept  2014. 

Numeric  Javascript,  version  1.2.6,  http://www.numericis.com,  accessed  22-Sept  2014. 

™  Sylvester:  Vector  and  Matrix  math  for  JavaScript,  version  0.1.3,  http://svlvester.icoglan.com,  accessed  22-Sept. 
2014. 

Backbone.js,  version  1.1.2,  http://backboneis.org,  accessed  20-Sept  2014. 
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by  overriding  the  getinput  method.  Similarly,  the  Smooth  class  defines  an  exponential 
smoothing  of  an  input  signal. 

The  Simulator  aggregates  Entity  objects  for  a  time-managed  simulation.  The  execute 
method  initializes  (init)  and  advances  (advance)  until  the  isComplete  method  returns 
true.  The  init  method  initializes  all  entities  at  the  specified  initTime  value  and  triggers  an 
"init"  event.  The  advance  method  tick/tocks  all  entities  with  the  specified  timeStep  value, 
advances  simulation  time,  triggers  an  "advance"  event,  and  triggers  a  "complete"  event  if 
complete.  The  default  isComplete  method  checks  the  current  simulation  time  against  the 
specified  maxTime  value. 

Ported  Design  Flow  Model 

The  Design  Flow  model  is  ported  to  JavaScript  by  defining  an  Application  class  which 
composes  a  Model  and  Simulator  object  as  illustrated  in  the  object  class  diagram  in  Figure 
40.  The  Model  class  includes  all  required  SD  components  to  replicate  the  Design  Flow  Model 
including  timers,  flows,  parameters,  stocks,  and  exponential  delay  and  smoothing  functions. 


Figure  40.  The  ISRM  Model  class  composes  many  Entity  objects. 

Additionally,  the  Model  class  includes  attributes  to  set  initial  parameter  values.  This  allows 
simple  model  parameter  changes  by  altering  the  constructor.  For  example,  two  models  may  be 
defined  by: 

var  metaOff  =  new  ISRM. Model ( {metaFlag:  0});  //  model  instantiation  without  META 
var  metaOn  =  new  ISRM. Model ( {metaFlag:  1});  //  model  instantiation  with  META 

Results  from  the  original  Vensim  Design  Flow  Model  and  the  new  JavaScript  ported  model  are 
cross-validated  by  comparing  outputs  at  each  time  step  under  several  inputs.  Numerical 
outputs  are  identical  between  the  two  models  for  the  no-META  condition.  Under  the  META 
condition  differences  are  observed  in  intermediate  time  periods  for  some  variables,  possibly 
due  to  undocumented  implementation  details  for  nested  functions  in  Vensim.  Despite  transient 
differences,  the  overall  impact  on  final  simulation  values  is  negligible  (<  0.03%  difference  for  all 
variables).  Additional  validation  should  identify  the  exact  implementation  of  nested  functions  in 
Vensim  to  resolve  discrepancies. 
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JavaScript  Model  Ul 


The  model  user  interface  (Ul)  uses  a  combination  of  HTML  and  CSS  to  structure  and  style  a  web 
page  and  JavaScript  for  behaviors  such  as  plotting,  click-and-dragging,  and  data  toggling  in  a 
standard  browser  environment.  Figure  41  shows  a  screen  capture  of  the  Ul.  The  top  section 
controls  simulations,  the  middle  section  plots  data,  and  the  bottom  section  visualizes  the  stock- 
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Figure  41.  Screen  capture  of  the  ISRM  user  interface. 


Several  JavaScript  libraries  contribute  to  the  overall  Ul.  First,  (Query  simplifies  interaction  with 
the  HTML  document  object  model  (DOM)  for  animations,  form  inputs,  and  event  handling. 
For  example,  the  text  inputs  and  buttons  at  the  top  of  the  screen  allow  the  user  to  set  initial 
and  final  simulation  times,  time  step  duration,  and  execute,  start,  stop,  and  reset  a  simulation 
execution.  Additional  buttons  allow  data  export  to  CSV  and  JSON  file  formats.  (Query  Ul 


jQuery,  version  2.0.3,  http://iquerv.com,  accessed  22-Sept  2014. 


79 


provides  user  interface  widgets. Dialog  boxes  edit  simulation  settings  and  provide  view  or 
edit  access  to  individual  model  components  including  parameters,  stocks,  and  flows. 

Plot  provides  graphical  plotting  capabilities. Customizable  plots  display  or  more  data  series 
during  a  simulation  execution.  The  plots  can  be  updated  in  simulation  time;  however,  the 
animation  process  is  resource-intensive  and  requires  about  18  seconds  per  120-month 
execution  with  0.25-month  time  steps  on  a  desktop  computer  with  the  Google  Chrome 
browser.  Alternatively,  plots  can  be  configured  to  update  only  at  the  end  of  the  simulation  to 
achieve  a  10-fold  simulation  speedup  (about  1.5  seconds  per  execution). 

Finally,  kinetic.js  provides  an  improved  object-oriented  canvas  environment. The  canvas 
element  replicates  the  stock-and-flow  diagram  from  the  original  Design  Flow  Model.  Users  click 
and  drag  stocks  (rectangles),  flows  (black  labels),  parameters  (blue  labels),  and  shadow 
variables  (gray  labels)  to  move  on  the  interface.  Double-clicking  any  field  opens  a  dialog  to  edit 
parameter  values,  view  flow  values,  view/edit  stock  values,  and  toggle  plotting. 

Limitations 

The  current  ISRM  application  is  limited  by  only  allowing  users  to  change  input  parameters  for  a 
fixed  model  structure.  Input  parameters  are  most  effective  to  toggle  flags  such  as  META 
features  or  change  generation.  They  can  also  customize  certain  factors  such  as  productivity, 
model  library  coverage  and  integrity,  and  staff  efficiency.  However,  input  parameters  cannot 
make  more  sophisticated  changes  to  the  model  behavior  to  alter  original  assumptions. 

There  are  a  number  of  assumptions  of  the  original  Design  Flow  Model  which  may  limit  its 
applicability  to  broader  engineering  projects.  For  example,  it  does  not  enforce  staffing  level 
constraints  for  design  processes.  It  also  assumes  a  ramp-up  profile  for  initial  requirements 
elicitation,  implications  of  complexity  for  design  productivity,  and  mechanics  of  change 
generation.  Changing  these  assumptions  requires  a  new  model-building  activity  rather  than  the 
current  model-using  activity. 

While  the  JavaScript  API  is  particularly  amenable  to  overriding  existing  definitions,  it  requires  a 
certain  level  of  familiarity  with  programming  and  the  JavaScript  language.  Furthermore,  the 
existing  Ul  only  displays  a  fixed  model  structure  and  cannot  automatically  generate  new 
diagrams.  Adding  model-building  activities  to  the  ISRM  would  require  a  significant  software 
development  effort  equivalent  to  creating  a  tool  such  as  Vensim  or  Insight  Maker  and  is 
considered  out-of-scope  at  this  time. 

Next  Steps 

Next  steps  include  improvements  to  the  Phase  1  ISRM  application  and  completion  of  the  Phase 
2  application  for  rapid  sensitivity  analysis  of  model  results. 

jQuery  Ul,  version  1.10.3,  http://iquervui.com,  accessed  22-Sept  2014. 

Plot,  version  0.8.1,  http://flotcharts.org,  accessed  22-Sept  2014. 

Kinetic.js,  version  5.1.0,  http://www.kineticis.com,  accessed  22-Sept  2014. 
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Initial  use  of  the  ISRM  Phase  1  application  identified  several  potential  improvements.  A  new 
alternative  model  view  will  add  Ul  widgets  to  edit  key  parameters  of  interest.  For  example, 
switch  widgets  toggle  META  or  change  generation  flags  and  slider  widgets  modify  other 
numeric  parameters  such  as  schedule  pressure,  component  model  library  integrity,  and 
novelty.  Improved  Ul  dialogs  will  add  more  descriptive  information  including  model 
assumptions  and  documentation  of  underlying  equations.  Finally,  new  parameters  will  be 
introduced  to  more  easily  adapt  the  model  to  new  contexts  without  editing  model  behaviors 
specified  in  flow  variables. 

Future  work  will  also  evaluate  model  performance  to  meet  needs.  Current  model  execution 
takes  about  1.5  seconds  on  desktop  computers.  If  this  performance  is  insufficient  for  Phase  2 
applications,  linear  algebra  or  vector  math  libraries  (such  as  Sylvester  or  Numeric.js  discussed 
previously)  may  help  parallelize  numeric  integration  and  achieve  performance  improvements. 
Phase  2  will  also  consider  server-side  JavaScript  execution  to  supplement  client-side  execution 
to  parallelize  model  simulation. 

Phase  2  development  activities  described  in  Section  3  will  explore  and  evaluate  technologies  to 
generate  and  interpret  large  datasets  generated  from  multiple  model  executions.  A  data 
storage  mechanism  will  be  crucial  to  avoid  rerunning  time-consuming  model  executions.  Non¬ 
relational  databases  will  be  evaluated  to  avoid  predefined  schemas.  Communication  between  a 
client  and  server  will  use  an  asynchronous  JavaScript  and  XML  (AJAX)  method  with  the 
representational  state  transfer  (REST)  architectural  style  for  web  services. 

Finally,  analysis  conducted  under  Phase  2  will  use  the  new  application  to  understand  the 
sensitivity  of  ISRM  results  to  various  input  parameters.  Initial  studies  will  focus  on  existing 
application  cases  developed  for  the  initial  Design  Flow  Model.  Past  sensitivity  analysis  was 
limited  to  three  input  parameters  levels  due  to  time  constraints,  however  improvements  will 
evaluate  greater  numbers  of  levels  for  each  parameter  of  interest  and  consider  interaction 
effects  between  parameters.  Results  will  demonstrate  the  capabilities  of  the  ISRM  including 
innovative  visualization  techniques  to  interpret  large  datasets  generated  across  many  model 
executions. 
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Moving  Forward  to  Phase  Two 


•  The  research  team  will  be  using  knowledge  and  information  gained  in  Phase  1  to  focus 
ongoing  efforts  in  Phase  2  to  further  explore  the  identified  IMCSE-related  considerations 
within  four  key  areas,  and  the  challenges  and  opportunities  at  their  intersection. 

•  The  pathfinder  workshop  plan  will  be  finalized  and  the  workshop  will  be  scheduled.  The 
workshop  will  be  held  and  a  workshop  report  will  be  published  and  released  to  elicit 
comments  and  recommendations.  Approaches  to  creating  a  broader  collaboratively-derived 
research  agenda  have  been  identified,  and  will  be  used  to  design  the  next  steps  in  building  a 
community  for  IMCSE  research. 

•  A  first  prototype  for  interactive  Epoch-Era  Analysis  will  be  completed  and  tested  with  a  case 
application,  along  with  preliminary  supporting  infrastructure,  which  will  then  be  used  to 
inform  the  design  of  a  next  version  prototype.  Specific  next  steps  are  described  on  page  47. 

•  The  team  will  continue  analysis  of  the  value  model  trades  in  the  demonstration  case,  along 
with  developing  a  more  complete  framework  and  process  for  how  to  conduct  value  model 
trades  more  generally.  Specific  next  steps  are  described  on  page  60. 

•  The  IVTea  Suite  will  continue  to  undergo  refinement  of  user  interface,  data  handling,  as  well 
as  development  of  additional  widgets  that  support  ongoing  research.  Specific  next  steps 
are  described  on  page  69. 

•  The  extended  interactive  schedule  reduction  model  prototype  will  be  completed  and  made 
available  for  user  testing.  Specific  next  steps  are  described  on  page  80. 

•  The  research  team  will  use  the  results  of  Phase  1  to  develop  several  publishable  papers  for 
the  CSER2015. 
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Transition  Objectives 


An  imperative  for  SERC  research  teams  is  the  effective  transition  of  research  to  practice, 
including  transfer  of  new  knowledge,  research  findings,  and  new  MPTs  to  members  of  the 
community  of  interest.  In  Phase  1,  we  have  developed  our  initial  plan  toward  this  objective.  The 
plan  includes  identifying  and  working  with  transfer  partners  and  research  collaboration 
partners.  Included  in  our  evolving  plan  are  the  following  objectives: 

•  IMCSE  research  on  current  state  of  the  art  and  practice  will  be  shared  among 
participants  in  a  planned  workshop  to  be  held  in  Phase  2,  and  in  subsequent  exchanges 
extending  from  this  workshop  via  teleconferences  and  meetings.  Workshop  attendees 
will  be  asked  to  identify  ways  in  which  the  community  of  interest  can  be  extended. 

•  The  Pathfinder  Project  Report  will  synthesize  the  findings  on  current  art  and  practice, 
and  define  a  way  forward  to  build  a  community  of  researchers.  This  report  will  be  issued 
at  the  end  of  Phase  2,  and  a  review/comment  period  will  follow  release  of  the  report. 

•  During  Phase  2,  the  research  team  will  develop  the  expanded  list  of  individuals  and 
organizations  to  be  contacted  for  inputs  for  the  activity  of  developing  a  collaboratively- 
derived  research  agenda.  A  working  paper  will  be  developed  during  Phase  2  to  capture 
the  approach  and  lessons  learned  in  creating  this  agenda,  as  well  as  the  results  of  this 
activity.  A  journal  submission  will  subsequently  be  prepared  from  the  working  paper. 

•  Results  of  the  Interactive  Schedule  Reduction  Model  (project  2)  and  Interactive  Epoch- 
Era  Analysis  (project  3)  will  be  shared  with  the  broader  SERC  community,  in  selected 
meetings  and  workshops,  such  as  INCOSE  IW15  and  CSER  2015.  A  paper  on  each  of 
these  projects  will  be  submitted  to  CSER  2015. 

•  At  the  end  of  Phase  2,  an  initial  demonstration  prototype  of  the  interactive  Epoch-Era 
Analysis  will  be  made  available  to  elicit  feedback.  A  plan  will  be  developed  to  further 
define  transition  of  the  prototype  for  user  testing.  At  the  end  of  Phase  2,  a  version  of 
the  interactive  schedule  reduction  model  prototype  will  be  made  available  for  user 
testing.  During  Phase  2,  a  webpage  will  be  created  to  provide  easy  access  to  prototypes, 
guidance  documentation  and  other  related  documents. 

•  Throughout  the  effort,  synergies  with  other  SERC  tasks  will  be  identified  and  leveraged 
to  transition/implement  resulting  capabilities  of  this  project,  as  well  as  to  provide 
relevant  information  to  impact  the  work  of  other  researchers. 
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Conclusions 


The  research  team  has  now  completed  the  Phase  1  effort,  taking  place  over  a  four  month 
period.  The  activities  within  the  three  thrusts  -  foundations,  fundamentals,  and  applications  - 
are  on-track  for  the  overall  goals  of  this  year's  research,  and  readiness  to  move  into  Phase  2  has 
been  achieved.  The  research  team  has  discovered  project-specific  challenges  and  opportunities 
in  the  broader  literature  and  community  of  interest.  Some  continued  divergent  investigation  is 
still  necessary,  and  by  the  end  of  Phase  2  the  team  expects  to  have  converged  on  research 
themes  and  questions.  The  pathfinder  workshop  will  play  an  important  role  in  this 
convergence,  and  will  result  in  a  workshop  report  aimed  to  inform  and  to  elicit  additional 
inputs.  The  research  team  expects  to  continue  its  efforts  in  Phase  2  to  complete  the 
development  of  a  user-testing  prototype  for  the  Interactive  Schedule  Reduction  Model.  The 
Interactive  Epoch-Era  Analysis  activity  is  progressing  according  to  plan,  and  the  team  expects  to 
complete  a  demonstration  prototype  at  the  end  of  Phase  2,  which  will  enable  the  team  to  gain 
specific  feedback  for  the  Phase  3  effort  to  result  in  a  pilot-test  version  of  the  prototype. 
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Appendix  A 


This  appendix  includes  a  short  summary  of  selected  relevant  prior  SERC  projects  we  have  identified  that 
inform  this  work.  The  research  team  is  in  the  process  of  identifying  points  of  connection  with  other 
SERC  projects. 

Graphical  Concepts.  In  related  prior  SERC  research  the  vision,  feasibility  assessment  and  initial  process 
for  a  Graphical  CONORS  development  environment  for  agile  systems  engineering  was  developed  (SERC- 
2009-TR-003“®).  This  research  investigated  current  approaches  to  Concept  of  Operations  (CONORS) 
development  in  use  in  various  DoD  and  commercial  organizations  with  the  goal  of  understanding  why 
CONORS  creation  is  such  a  lengthy  process,  and  how  the  process  can  be  made  more  agile.  Based  on 
findings,  an  agile  CONORS  process  that  emphasizes  stakeholder  involvement  and  expedites  shared 
mental  models  development  is  put  forth  by  the  team.  An  initial  prototype  was  developed  toward  a 
concept  engineering  software  demonstrator  that  enabled  soldiers  to  develop  a  limited  set  of  scenarios 
centered  on  squad  operations  (SERC-2012-TR-030“^).  The  prototype  was  then  extended  as  3D  virtual 
guide  intended  to  assist  one  assigned  to  CONORS  development,  through  the  setup  of  a  combat  scenario 
and  the  use  of  the  Integrated  CONORS  Environment  Framework  (ICEF).  This  research  (SERC-2013-TR- 
031-2“®)  demonstrated  that  it  is  possible  to  utilize  the  strength  of  a  3D  game  development  environment 
to  create  a  graphical  CONORS  creation  tool  that  is  easy  for  a  soldier  to  use. 

Defense  Architecture  Framework.  Different  Department  of  Defense  communities  prepare  models  for 
architecture  compliance  (e.g.,  to  maintain  JCIDS  requirements),  for  simulation  purposes  (e.g.,  for 
performance  estimates)  and  software  engineering  (e.g,  for  model-based  code  generation).  Little,  if  any, 
information  transfer  and  model  reuse  takes  place  across  these  communities  of  interest,  which  leads  to 
redundant  efforts,  models  that  are  out  of  sync,  and  lost  domain  knowledge.  Differences  in  methods, 
tools,  and  data  formats  are  a  major  reason  for  this  disconnect.  The  charter  of  RT-24  was  to  investigate 
mechanisms  that  could  help  bridge  the  divide  between  the  modeling  &  simulation,  software 
engineering,  and  enterprise  architecture  modeling  communities^®. 

System  2020  Strategic  Initiative.  Systems  2020  is  the  research  effort  to  answer  a  ajor  portion  of  the 
challenge  embodied  in  the  DoD's  science  and  technology  priority  for  Engineered  Resilient  Systems  (ERS). 
As  a  follow-on  to  the  SERC's  work  in  defining  technical  approaches  for  Systems  2020,  DASD(SE) 
requested  the  SERC  to  work  on  two  tasks.  Task  1  involved  working  with  Government  research 
and  engineering  centers,  and  laboratories  to  characterize  the  design  and  systems  engineering  (SE)  tools 
available  to  DoD  projects,  along  with  their  potential  for  using  these  tools  in  integrated  demonstrations 
of  their  capability  to  support  representative  future  DoD  systems  acquisitions  with  respect  to  purpose, 
affordability,  and  interoperability.  Task  2  involved  identifying  several  design  challenge  problems  to 


SERC-2009-TR-003,  Cloutier,  R.  and  Mostashari,  A.,  Investigation  of  a  Graphical  CONORS  Development 
Environment  for  Agile  Systems  Engineering,  Systems  Engineering  Research  Center,  Final  Technical  Report,  October 
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characterize  the  integrated  environment  capabilities  being  identified  in  Task  1.  Based  on  discussion  of 
the  Task  1  analysis  results  with  the  sponsors,  the  original  Task  2  statement  was  reinterpreted  to  involve 
the  SERC  Research  Council  in  defining  one  or  more  representative  future  design  challenge  problems,  and 
in  determining  key  research  ideas  and  directions  that  would  enable  DoD  to  cope  with  the  challenges. 
This  report  includes  the  resulting  two  Grand  Challenge  scenarios  of  particularly  difficult  threat 
complexes  beyond  the  reach  of  current  tool  support  capabilities,  with  indications  of  the  type  of  next- 
generation  tools  that  would  enable  successful  DoD  responses.  It  also  presents  four  high-leverage 
research  areas  that  would  be  key  to  realizing  the  rapid  and  effective  results  described  in  the  scenarios, 
using  the  Heilmeier  criteria  for  evaluating  proposed  research  initiatives:  (1)  Affordability,  Agility,  and 
Resilience;  (2)  Enterprise  Systems  Engineering  and  Model  Integration;  (3)  Trusted  Systems  and  Cyber 
Security;  and  (4)  Human-Determined  Systems^“. 

Introducing  Model  Based  Systems  Engineering  Transforming  System  Engineering  through  Model- 
Based  Systems  Engineering.  This  research  topic  (RT-48)  focuses  on  a  Vision  held  by  NAVAIR's 
leadership  to  assess  the  technical  feasibility  of  creating/leveraging  a  more  holistic  Model-Based  Systems 
Engineering  (MBSE)  approach.  The  expected  capability  of  such  an  approach  would  enable  mission-based 
analysis  and  engineering  that  reduces  the  typical  time  by  at  least  25  percent  from  what  is  achieved 
today  for  large-scale  air  vehicle  systems.  The  research  need  includes  the  evaluation  of  emerging  system 
design  through  computer  (i.e.,  digital)  models.  The  first  phase  of  the  effort  began  investigating  the 
technical  feasibility  of  moving  to  a  "complete"  model-driven  lifecycle  and  includes  four  key  tasks  that 
can  be  summarized  as  follows:  Surveying  Industry,  Government  and  Academia  to  understand  the  state- 
of  the-art  of  a  holistic  approach  to  MBSE;  Develop  a  common  lexicon  for  MBSE,  including  model  types, 
levels,  uses,  representation,  visualizations,  etc.:  Model  the  "Vision,"  but  also  relate  it  to  the  "As  Is" 
process;  Integrate  a  Risk  Management  framework  with  the  Vision  This  report  provides  details  about 
each  task,  the  focus  of  the  research  questions,  accomplishments,  and  the  plans  for  the  Phase  II  efforts, 
which  are  to  continue  under  another  RT. 

Transforming  Systems  Engineering  through  Model  Based  Systems  Engineering^^^.  The  objective  of  the 
RT-46  research  effort  is  to  assess  whether  it  is  technically  feasible  to  transform  systems  engineering 
using  a  fully  formalized  Model-Based  Systems  Engineering  (MBSE)  approach  (i.e.  model-centric 
engineering)  with  integrated  and  interoperable  models  to  associated  simulations,  surrogates,  digital 
assets,  and  supporting  environment.  Application  is  to  be  tested  in  NAVAIR.  According  to  the  report, 
"the  on-going  research  will  be  informed  by  the  application  of  the  results  and  then  the  research  will  be 
tailored  to  newly  discovered  needs  going  forward.  This  should  enable  NAVAIR  to  create  a  plan  informed 
by  the  research  results  for  implementation  relatively  early  in  the  results  cycle". 

Integration  of  M&S  (Modeling  and  Simulation),  Software  Design  and  DoDAF  (Department  of  Defense 
Architecture  Framework  (RT  24).^^^  Different  Department  of  Defense  communities  prepare  models  for 
architecture  compliance  (e.g.,  to  maintain  JCIDS  requirements),  for  simulation  purposes  (e.g.  for 
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performance  estimates)  and  software  engineering  (e.g,  for  model-based  code  generation).  Little,  if  any, 
information  transfer  and  model  reuse  takes  place  across  these  communities  of  interest,  which  leads  to 
redundant  efforts,  models  that  are  out  of  sync,  and  lost  domain  knowledge.  Differences  in  methods, 
tools  and  data  formats  are  a  major  reason  for  this  disconnect.  The  charter  of  RT  24  was  to  investigate 
mechanisms  that  could  help  bridge  the  divide  between  the  modeling  &  simulation,  software 
engineering,  and  enterprise  architecture  modeling  communities. 

Tradespace  and  Affordability  (Task  46,  113)“^  The  focus  of  the  second  phase  (RT  46)  and  third  phase 
(RT  113)  is  to  apply  the  methods  and  tools  developed  in  the  first  phase  on  problems  relevant  to  DoD, 
ideally  using  the  information  available  from  development  of  a  large  weapon  system,  or  a  large 
automated  information  system.  Ideally,  the  SERC  will  work  with  the  system  developer  to  gain  a  deep 
understanding  of  the  strengths  and  limitations  of  the  tradespace  tools  methods  developed  under  Phase 
1.  Phase  2  and  3  activities  will  expand  the  set  of  ilities  represented  in  the  tradespace.  The  information 
learned  from  Phases  2  and  3  will  be  used  to  improve  the  frameworks  and  tools  developed  in  the  Phase  1 
activities. 

Engineered  Resilient  Systems:  Tradespace  Tools  (Task  120)“^  This  research  proposes  the  development 
of  an  analytical  construct  to  forecast  resilient  options  and  a  tradespace  toolset  framework  architecture 
in  support  of  Engineered  Resilient  Systems  (ERS).  This  includes  conducting  research  and  development  of 
methodologies  to  include  Needs  Context  based  Utility  Functions,  and  risk  mitigation  of  uncertain  future 
events  through  option  buy-ins.  Next,  this  effort  investigates  various  toolset  usability  upgrades,  building 
on  Georgia  Tech's  experience  in  building  web-based,  collaborative  systems  engineering  frameworks,  in 
light  of  related  stakeholder  requirements  and  use  cases.  Finally,  this  work  will  explore  a  series  of  ERS 
Architecture  tradespace  toolset  concepts  which  would  be  co-developed  between  Georgia  Institute  of 
Technology,  the  US  Army  Engineer  Research  and  Development  Center  (ERDC),  and  related  stakeholders. 

Development  and  Application  of  FACT  to  Support  USMC  Ground  Vehicle  Design  Analysis  (Task 

USMC  has  been  developing  the  Framework  for  Assessing  Cost  and  Technology  (FACT)  since  2011.  In  that 
time,  FACT  has  been  applied  to  multiple  ground  vehicle  platforms,  each  time  improving  the  Framework 
based  on  the  use  required  by  the  design  analysis  team.  In  this  task  researchers  will  improve  upon  FACT 
to  prepare  for  the  next  USMC  large  ground  vehicle  acquisition  program,  to  include  integration  of  new 
modeling  tools  and  simulation  environments,  toolset  improvements  such  as  a  task  queue  and  modular 
architecture  to  ease  collaborative  development,  and  improved  interactive  data  visualizations. 
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